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OPINION 


THINK  TANKS  AND 
THE  NATiONAl  SECURITY  STRATEGY 
FORMUIATION  PROCESS: 

4  COMPARISON  Of  CURRENT 
AMERICAN  AND  FRENCH  PAUERNS 


Robert  Ranquet 

This  essay  investigates  the  process  by  which  national  security  strategy  is 
formulated;  more  precisely,  it  will  look  at  the  specific  input  to  this  process  from 
the  organizations  known  as  think  tanks.  It  will  also  attempt  to  compare  the 
ways  think  tanks  influence  the  national  security  strategy  formulation  process 
in  the  United  States  and  in  France. 


We  did  not  have  to  wait  long  after 
the  collapse  of  the  Soviet  Union 
to  see  the  so-called  New  World 
Order  both  illustrated  and  challenged  by 
very  different  experiences — the  over¬ 
whelming  coalition  military  victory  in  the 
Persian  Gulf  on  one  hand,  and  the  piti¬ 
able  procrastination  of  Western  countries 
to  cope  with  the  Yugoslavian  crisis  on  the 
other — send  us  contradictory  messages 
that  we  must  carefully  decipher.  The  eu¬ 
phoria  resulting  from  the  former  may  well 
be  as  misleading  as  the  acrimony  we  see 
arising  from  the  latter. 

As  a  result  of  this  turmoil,  formulating 
national  security  strategy  has  become  a 
much  more  difficult  and  subtle  exercise 
than  it  was  under  the  traditional  strategy 


of  containment.  The  policymaker  now  has 
to  integrate  many  different  perspectives 
to  get  a  better  grasp  of  this  increasingly 
complex  art.  Our  investigation  of  the  na¬ 
tional  security  strategy  process  will  use, 
as  examples,  two  documents  that  were  is¬ 
sued  within  a  few  months  of  each  other. 
The  first  is  the  White  House  report  to  the 
Congress  entitled:  “A  National  Security 
Strategy  of  Engagement  and  Enlarge¬ 
ment'’  (July  1994).  The  French  govern¬ 
ment  issued  in  March  1994  its  “Livre 
Blanc  sur  la  Defense,”  which,  despite 
some  meaningful  differences  from  the 
White  House  report,  shows  many  signifi¬ 
cant  similarities  to  it.  This  white  paper  will 
therefore  be  used  as  an  example  of  the 
practice  on  the  French  side. 


1 


Acquisition  Review  Quarteriy — Winter  1997 


A  nation’s  national  security  strategy  has 
a  profound  impact  on  its  defense  acquisi¬ 
tion  policies,  not  only  in  determining  the 
types  of  systems  procured  and  quantities 
(i.e.  more  mobility  of  forces  versus  less 
heavy  defensive  weaponry),  but  in  the 
emphasis  the  nation  places  upon  coopera¬ 
tive  acquisition  of  defense  equipment  with 
its  allies  and  friends,  which  results  in  an 
acquisition  policy  that  favors  cooperative 
developments  of  a  new  defense  system. 
According  to  the  Department  of  Defense 
Directive  on  Defense  Acquisition,  even 
more  preferred  to  a  new  development  is 
the  procurement  (including  modification) 
of  commercially  available  systems  or 
equipment,  additional  production  (includ¬ 
ing  modification)  of  already  developed 
U.S.  military  systems  or  equipment,  or 
allied  systems  or  equipment. 

In  a  similar  way,  the  1994  French 
“Livre  Blanc  sur  la  Defense”  strongly 
stated  that:  “No  future  major  program  on 
conventional  weapons  seems  likely  to  es¬ 
cape  the  logic  of  (international)  coopera¬ 
tion.”  This  is  clearly  seen  as  a  key  factor 
in  affording  the  future  defense  expendi¬ 
tures  required  to  fulfill  future  military 
needs.  Therefore,  we  should  see  strong 
pressure  coming  from  acquisition  commu¬ 
nities,  on  both  sides  of  the  Atlantic,  to 
make  every  possible  effort  to  bring  to¬ 
gether  more  cooperative  projects. 

The  methodology  for  this  research  has 
included  contacts  with  numerous  and  vari¬ 
ous  think  tanks  both  in  Washington,  DC, 
and  in  Paris,  interviews  with  senior  po¬ 


litical  science  and  foreign  policy  analysts 
in  some  of  these  organizations,  and  a  gen¬ 
eral  survey  of  the  national  security  strat¬ 
egy  inputs  from  think  tanks  from  1992 
through  mid- 1994. 

The  Formulation  Products 


As  products  of  national  security  strat¬ 
egy  formulation  processes,  the  White 
House  report  will  be  refered  to  as  “NSS 
94,”  and  the  “Livre  Blanc  sur  la  Defense” 
as  the  white  paper. 

NSS  94  is  required  by  law:  The  demand 
for  this  document  originates  in  the 
Goldwater-Nichols  Act  of  1986,  Its  ob¬ 
jective  and  content  are  codified  in  50  USC 
404a  (see  Appendix  A).  This  report  must 
each  year  assess  the  general  frame  for  the 
national  security  strategy.  Presumably  it 
may  also  be  useful  in  supporting  the  an¬ 
nual  presidential  budget  request  to  the 
Congress.  Actually,  the  last  issues  of  the 
report  have  been  affected  by  several 
changes  in  the  administration,  and  have 
not  met  the  requirement  for  an  annual  is¬ 
sue.  As  high-level  political  documents, 
such  reports  are  not  intended  to  assess  pre¬ 
cisely  the  detailed  goals  and  means  of  the 
U.S.  policy,  but  are  limited  to  a  broader 
view  of  the  global  national  security  strat¬ 
egy.  It  is  up  to  more  focused  processes, 
such  as  the  Bottom-Up  Review  in  the  mili¬ 
tary  arena,  to  enter  into  this  level  of  de¬ 
tailed  information. 


Captain  (Armament  Corps)  Robert  Banquet  wrote  this  article  as  part  of  his  academic  re¬ 
quirements  at  the  Industrial  College  of  the  Armed  Forces  (National  Defense  University)  in  1994- 
95.  He  Is  presently  assigned  to  the  French  Embassy  in  Washington,  DC,  as  the  Attache  for 
Armaments. 
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The  French  white  paper  has  the  same 
broad  objective  of  assessing  the  national 
security  strategy.  Because  of  the  different 
balance  of  power  between  the  executive 
and  legislative  branches  of  government  in 
France,  this  white  paper  was  written  as  an 
initiative  of  the  prime  minister,  without 
any  formal  preset  frame.  Its  official  pur¬ 
pose  is,  as  stated  in  its  introduction:  “To 
acquire  a  better  understanding  of  our  time 
and  of  the  part  played  in  it  by  the  defense 
of  our  country.  To  place  defense  policy  in 
the  long-term  perpective  that  is  indispens¬ 
able  to  it.  To  explain  defense  to  the  French 
people  and  rally  their  support”  (p.  4).  It  is 
interesting  to  note  that  the  previous  white 
paper  on  defense  had  been  issued  22  years 
previously.  This  long  interval  between  the 
two  may  be  explained  by  the  relative  sta¬ 
bility  of  the  global  vision  of  the  world 
during  the  Cold  War  period.  It  also  prob¬ 
ably  reflects  a  traditional  French  consen¬ 
sus  on  the  national  strategic  posture 
throughout  the  internal  political  changes 
of  the  period:  the  eventual  adjustments  to 
the  strategy  were  made  during  periodic 
updates  of  the  successive  Military  Mid¬ 
term  Planning  Acts.  But,  the  dramatic 
changes  that  occurred  after  the  collapse 
of  the  Soviet  Union  obviously  required  an 
indepth  re-examination  of  the  national 
security  strategy. 

The  purpose  of  the  white  paper  is  not 
primarily  to  inform  the  Parliament,  but  to 
achieve  a  broader  education  of  public 
opinion.  Like  NSS  94,  it  has  embedded  a 
much  larger  concept  of  national  security, 
departing  from  the  merely  military  ap-^ 
proach  that  prevailed  a  few  years  ago. 
Unlike  NSS  94  and  more  like  the  Bottom- 
Up  Review,  it  takes  in  fairly  detailed  con¬ 
sideration  the  military  forces  format.  Be¬ 
yond  these  technical  distinctions,  we  can 


also  note  differences  that  deal  more  with 
differences  in  the  approach  of  the  secu¬ 
rity  concept  itself.  For  instance,  NSS  94 
pays  special  attention  to  the  doctrine  for 
the  engagement  of  U.S.  forces  abroad,  re¬ 
flecting  a  major  enduring  concern  among 
the  American  people. 

It  is  also  significant  to  note  that  NSS 
94  discusses  environmental  issues,  a  sub¬ 
ject  absent  from  the  French  white  paper. 
But,  the  white  paper  does  discuss  socio¬ 
cultural  issues,  such  as  the  relationship 
between  the  nation  and  its  military  insti¬ 
tution,  that  are  absent  from  NSS  94.  It 
would  be  interesting,  but  also  beyond  the 
purpose  of  this  essay,  to  investigate  fur¬ 
ther  the  differences  in  the  content  of  the 
two  documents  (see  Appendix  B):  Let  us 
simply  suggest  here  that  these  differences 
are  more  dependent  on  contingent  domes¬ 
tic  political  considerations  at  the  time  of 
their  writing,  than  on  more  fundamental 
features  of  the  so-called  national  charac¬ 
ters. 


The  Formulation  Processes 


The  U.S.  Process 

NSS  94,  like  its  predecessors,  was  is¬ 
sued  as  the  product  of  an  interagency  pro¬ 
cess,  in  which  the  National  Security  Coun¬ 
cil  (NSC)  plays  the  central  role.  A  few 
senior  officers  in  this  organization  were 
in  charge  of  producing  the  drafts,  and  of 
acting  as  the  linchpin  of  the  process.  The 
drafting  began  in  July  1993,  based  on  the 
strategic  framework  that  was  used  in  par¬ 
allel  for  the  Department  of  Defense  (DoD) 
Bottom-Up  Review.  President  Bill 
Clinton,  who  had  been  in  office  for  six 
months,  made  it  clear  that  the  document 
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should  also  integrate  some  aspects  from 
his  own  campaign  platform.  The  first  draft, 
ready  by  August  1993,  was  not  circulated 
at  once  among  the  different  agencies,  but 
was  used  as  a  base  for  six  major  speeches 
on  foreign  policy  that  were  delivered  by 
the  president  and  his  cabinet  members  in 
September. 

From  the  comments  they  received  on 
these  speeches,  the  NSC  team  produced  a 
second  draft,  which  was  submitted  to  the 
other  agencies  by  the  end  of  October.  Then 
at  year’s  end,  DoD  was  to  submit  its  bud¬ 
get  request  to  the  White  House.  It  became 
apparent  that  the  two  processes — budget¬ 
ing  and  national  security  strategy  formu¬ 
lation  (which  until  then  had  operated  sepa¬ 
rately) — had  to  be  joined.  So  a  third  draft 
was  prepared,  reflecting  the  changes  that 
then  affected  the  strategic  thinking  of 
some  decision-making  layers  at  the  White 
House  and  at  the  Pentagon, 

Since  it  appeared  unlikely  that  unani¬ 
mous  agreement  would  be  achieved  on 
this  third  version,  a  reduced  team  drafted 

a  fourth  version 


"Unlike  the  (theo¬ 
retically)  annual  U.S* 
report,  the  French 
white  paper  was  a 
rather  exceptional 
event,  and  its  elabo¬ 
ration  followed  a 
special  ad  hoc  pro¬ 
cess." 


in  March  1994, 
which  eventu¬ 
ally  was  ap¬ 
proved  and  is¬ 
sued  in  July.  Its 
publication,  ac¬ 
cording  to  its 
writers,  seems 
to  have  passed 
largely  unno¬ 


ticed,  except  perhaps  by  a  very  special¬ 


ized  public. 


The  process  took  one  full  year,  during 


which  numerous  interactions  with  other 


government  processes  had  introduced 
painful  iterations.  This  was  probably  not 
optimal  for  a  process  that  must  be  repeated 


yearly.  Nevertheless,  we  must  recognize 
that  NSS  94  was  the  first  formulation  by 
a  Democratic  administration  of  its  national 
security  strategy  after  the  collapse  of  the 
Soviet  Union,  in  a  context  of  already  de¬ 
clining  military  budgets  at  home,  and  sev¬ 
eral  embarrassing  situations  (e.g.,  Soma¬ 
lia  and  Bosnia)  developing  abroad.  Such 
may  explain  the  difficulty. 

The  French  Process 

Unlike  the  (theoretically)  annual  U.S. 
report,  the  French  white  paper  was  a  rather 
exceptional  event,  and  its  elaboration  fol¬ 
lowed  a  special  ad  hoc  process.  In  this 
case,  the  prime  minister  assigned  the  task 
on  May  26, 1993,  to  a  special  interagency 
commission  chaired  by  the  vice-chairman 
of  the  Conseil  d’Etat,*  which  gathered  20 
high-ranking  officials  from  the  different 
concerned  ministries,  and  4  independent 
experts,  among  whom  were  the  chairman 
of  the  French  Institute  for  International 
Relations  (IFRI)  and  the  honour  chairman 
of  the  Renault  automobile  firm  (see  Ap¬ 
pendix  C).  This  commission  worked  out 
its  final  document  through  the  usual  pro¬ 
cess  of  breaking  down  into  several  sub¬ 
commissions,  addressing  issues  such  as: 
(a)  the  global  outlook  and  the  strategic 
trends,  (b)  Europe  and  Defense,  (c)  finan¬ 
cial  resources,  (d)  industrial  base,  and  (e) 
human  resources. 

Meanwhile,  the  Ministry  of  Defense 
(MoD)  itself  organized  its  contribution 
through  diverse  working  groups  that  fed 
the  subcommissions  with  information  and 
propositions  that  reflected  the  MoD’s  pro¬ 
spective. 

These  groups  were  able  to  incorporate 
numerous  inputs  from  nongovernment 
sources,  such  as  the  major  companies  of 
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the  defense  industry.  All  this  work  was 
concluded  within  nine  months,  with  a  130- 
page  document  that  received  a  double 
preface  from  both  the  prime  minister  and 
the  minister  of  defense.  The  document  was 
approved  on  February  16,  by  President 
Francois  Mitterand  after  a  discussion 
within  the  “Conseil  de  Defense,”^  and  was 
presented  to  the  defense  committees  of  the 
two  Parliament  chambers  on  February  23. 
On  the  same  day,  the  document  was  pub¬ 
licized  in  the  media.  As  a  white  paper,  this 
document  did  not  have  to  be  submitted  for 
the  Parliament’s  approval.  Moreover,  it 
was  intended  to  provide  a  broad  strategic 
framework  for  the  future  government 
work,  and  more  specifically  for  the  exami¬ 
nation  and  enactment  by  the  Parliament 
of  the  Military  Midterm  Planning  Act 
(1995-2000  period)  that  was  to  be  sub¬ 
mitted  at  the  1994  fall  session  of  the  Par¬ 
liament, 


Think  Tanks  ih  the  American  Process 
The  American  Think  Tanks 

It  is  not  the  purpose  of  this  work  to  in¬ 
vestigate  in  depth  the  nature  of  the  Ameri¬ 
can  think  tanks.  Nevertheless,  it  is  strik¬ 
ing  to  recognize  that  any  definition  has 
difficulties  embracing  the  variety  of 
groups,  companies,  organizations,  etc., 
that  call  themselves  think  tanks.  James  A. 
Smith  has  defined  them  as:  “...[the]  plan¬ 
ning  and  advisory  institutions...,  the  pri¬ 
vate,  nonprofit  research  groups  that  oper¬ 
ate  on  the  margins  of  this  nation’s  formal 
political  processes”  (Smith,  1993,  p.  xiii). 
Although  this  definition  captures  the  es¬ 
sential  features  of  most  think  tanks,  some 
significant  organizations  do  not  fit  it,  but 


are  nevertheless,  without  any  doubt,  real 
think  tanks.  For  example,  they  include  the 
Institute  for  National  Security  Studies, 
which  is  actually  part  of  DoD,  the  Armed 
Forces  Journal  International,  which  be¬ 
longs  more  to  the  media  world,  or  Rand  Cor¬ 
poration,  which  has  many  features  of  a  “nor¬ 
mal”  advising  and  servicing  company. 

I  would  suggest  here  that,  more  than 
by  their  actual  legal  form,  fiscal  status, 
private  or  public  ownership,  or  origin  of 
funding,  think  tanks  are  characterized  by 
their  common  purpose  to  influence  politi¬ 
cal  processes  from  the  margins.  The  time 
horizon  of  their  influence  and  their  indi¬ 
vidual  strategy  to  achieve  this  influence 
may  vary  from  one  to  another,  but  all  think 
tanks  pursue  that  kind  of  purpose,  and  op¬ 
erate  on  the  margin  of  the  formal  process, 
or,  more  precisely,  at  the  junction  of  this 
process  with  some  specific  outside  world. 
They  play  an  indispensable  role  in  formu¬ 
lating  both  the  questions  and  the  answers 
in  the  dialogue  between  two  worlds:  the 
world  of  informed  public  opinion  and  its 
experts,  and  the  world  of  governmental 
bureaucracies.  In  this  dialogue,  where, 
more  and  more  “...  the  most  serious  ques¬ 
tions  cannot  even  be  posed,  let  alone  an¬ 
swered,  in  the  language  of  common 
sense,”  they  act  as  intermediaries  and  in¬ 
terpreters  (Smith,  1993,  p.  238).  They 
both  feed  the  bureaucracy  with  the  ideas 
that  bureaucrats  have  no  time  to  dig  out 
by  themselves  (ideas  brokers,  as  Smith  de¬ 
scribes  them),  and  to  circulate  these  ideas 
to  build  public  consensus.  They  “...  are  the 
principal  arteries  through  which  knowl¬ 
edge  flows  and  is  absorbed,  like  oxygen, 
into  the  bloodstream  of  political  life” 
(Smith,  1993,  p.  238). 
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What  Were  the  Issues  Raised, 

AND  How? 

The  Role  of  the  United  States  in  the 
World.  The  fundamental  issue  that  must 
be  addressed  in  the  formulation  of  a  na¬ 
tional  security  strategy  is  what  this  nation 
sees  as  its  role  in  the  world.  This  has  been 
and  still  is  a  very  controversial  issue  in 
the  United  States.  The  classic  opposition 
of  the  isolationist  view  versus  the  globalist 
view  has  been  raised  again  in  the  after- 
math  of  the  Cold  War,  with  shifting  em¬ 
phasis  on  the  economic,  social,  or  cultural 
aspects  of  the  debate.  More  precisely,  the 
debate  has  shifted  from  whether  or  not  the 
United  States  should  withdraw  to  its  con¬ 
tinent  and  concentrate  on  domestic  prob¬ 
lems,  to  what  precise  National  interests  are 
at  stake  abroad  and  to  how  much  this  na¬ 
tion  is  willing  to  pay  for  the  preservation 
of  these  interests.  Think  tanks  have  widely 
discussed  this  issue.  The  conservative 
Heritage  Foundation  has  traditionally  ad¬ 
vocated  less  exclusive  American  engage¬ 
ment,  as  assessed,  for  example,  by  Doug 
Seay  (1992): 

America  need  not,  and  will  not  long 
wish  to,  continue  to  assume  the  prin- 
cipal  burden  for  keeping  order 
around  the  world.  But  it  does  have 
an  interest  in  the  maintenance  of  that 
order.  Only  by  encouraging  its  allies, 
past  and  future,  to  assume  their 
proper  share  of  the  burden  can  it 
safely  relinquish  the  lion’s  share  of 
the  burden. 

At  the  same  time,  the  Democratically 
oriented  Brookings  Institution  proposed  a 
more  assertive  stance  toward  global  en¬ 
gagement,  by  promoting,  for  example. 


William  Perry’s  concept  for  a  cooperative 
security: 

A  cooperative  security  regime  is  de¬ 
signed  to  minimize  any  underlying 
military  causes  for  such  conflicts,  to 
deter  rogue  nations  from  initiating 
such  conflicts,  and  if  deterrence  fails, 
to  provide  a  multinational  military 
force  to  defeat  any  aggressor  nation. 

From  the  libertarian  side  of  the  politi¬ 
cal  continuum,  the  CATO  Institution 
warns  against  any  American  involvement 
in  what  are  seen  as  outside  problems,  thus 
bringing  to  the  United  States  undue  risks 
and  excessive  government  burden 
(Ra  venal,  1991): 

In  the  emerging  era  of  international 
relations,  even  great  nations — even 
the  “sole  surviving  superpower,”  if 
one  insists  on  that — will  do  better  to 
adjust  to  the  conditions  of  the  inter¬ 
national  system  than  to  perpetuate  at¬ 
tempts,  however  attractive  and  ap¬ 
parently  constructive,  to  control  the 
course  of  events  in  the  world. 

A  significant  majority  agrees  on  the 
necessity  of  promoting  regional  solutions 
as  an  alternative  to  U.S.  intervention 
(Conry,  1994),  by  transferring  more  re¬ 
sponsibilities  to  regional  powers  and  re¬ 
lying  more  on  regional  security  structures. 
Institute  for  National  Security  Studies  se¬ 
nior  fellow  Patrick  M.  Cronin  argues  that 
“the  best  course  is  to  pursue  U.S.  inter¬ 
ests  internationally  through  a  concert  of 
power  with  our  key  allies,”  in  a  Wilsonian 
reminiscence  (Cronin,  1993). 

All  these  ideas,  the  contradiction  of 
which  is  at  the  core  of  the  most  existential 
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problem  facing  the  United  States,  are  ech¬ 
oed  in  NSS  94: ...  it  is  clear  that  we  cannot 
police  the  world;  but  it  is  equally  clear  that 
we  must  exercise  global  leadership”  (p,  5). 

The  National  Interests.  Even  if  “glo¬ 
bal  leadership”  is  a  generally  well-ac¬ 
cepted  (appealing  yet  vague)  American 
ideal,  it  is  hard  to  deduce  from  NSS  94 
what  precisely  are  the  national  interests 
that  we  are  protecting  by  exercising  this 
global  leadership,  especially  at  the  even¬ 
tual  cost  of  American  lives.  Think  tanks 
have  brought  some  tentative  answers  to 
this  question. 

Some  answers  are  in  the  negative  form: 
“Instead  of  assuming  grave  risks  when 
vital  American  interests  are  not  at  stake, 
the  United  States  should  distance  itself 
from  regional  disputes  that  could  go 
nuclear,”  the  CATO  Institute  warns 
policymakers  (Carpenter,  1993,  p.  1). 
Some  Wilsonian  interests,  which  had  be¬ 
come  favorites  of  American  foreign  policy 
in  past  admistrations,  do  not  appeal  to  the 
Heritage  Foundation:  “America  does  not 
have  endless  resources  to  squander  on 
some  open-ended  crusade  for  democracy 
and  human  rights”  (Holmes,  1994,  p.  7), 

Some  conclusions  are  more  assertive: 
Policymakers  should  classify  the  national 
interests  in  thinkable  categories,  such  as 
“vital,  important,  and  marginal,”  as  pro¬ 
posed  by  the  Heritage  Foundation 
(Holmes,  1994).  This  approach,  also 
stressed  by  military-academic  institutions 
such  as  the  National  Defense  University, 
may  be  too  rational  for  the  accommoda¬ 
tion  of  political  flexibility.  This  may  be 
why  NSS  94  prefers  to  focus  on  broader 
objectives,  such  as  “Enhancing  Our  Se¬ 
curity,”  “Promoting  Prosperity  at  Home,” 
and  “Promoting  Democracy,”  without 


paying  too  much  attention  to  the  real  na¬ 
ture  of  the  objective  links  between  these 
generic  goals  and  the  actual  national  se¬ 
curity  concerns.  One  could  argue  that 
these  broad  objectives  would  allow  any 
sort  of  foreign  or  domestic,  economic  or 
military  issue  the  government  deemed  im¬ 
portant  to  be  called  a  “national  security” 
issue.  As  a  matter  of  fact,  NSS  94  em¬ 
braces  a  very  wide  range  of  preoccupa¬ 
tions  in  the  generic  category  of  national 
interest:  For  example,  a  strong  emphasis 
in  the  document  is  put  on  environmental 
issues,  echoing  M.  Renner  of  the 
WorldWatch  Institute: 


Environmental  threats  with  the  po¬ 
tential  to  erode  the  habitability  of  the 
planet  are  forcing  humanity  to  con¬ 
sider  national  security  in  far  broader 
terms  than  that  guaranteed  solely  by 
force  of  arms  (Renner,  1989,  p.  7). 


The  Resources.  At  the  same  time,  this 
broad  picture  allows  NSS  94  to  remain 
very  elusive  about  what  should  be  the  ap¬ 
propriate  an¬ 


swer  to  a  threat 
directed  against 
these  interests. 
This  may  be,  af¬ 
ter  ail,  good 
strategy,  ac¬ 
cording  to  Sun 
Tsu’s  aphorism: 
“All  warfare  is 
based  on  decep¬ 
tion”  (1963,  p. 
66).  And  it  also 


"But  although  the 
document  prudently 
avoids  any  precise 
assessment  of  the 
effective  forces 
format  that  would 
meet  the  require¬ 
ments  of  the  strat¬ 
egy,  it  wisely  points 
at  the  generic  source 
of  any  national 
power:  the 
economy." 


^reserves  the 

possibility  for  any  necessary  adjustment 
pf  the  policy  in  the  difficult  art  of  resource 
illocation.  But  although  the  document 
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prudently  avoids  any  precise  assessment 
of  the  effective  forces  format  that  would 
meet  the  requirements  of  the  strategy,  it 
wisely  points  at  the  generic  source  of  any 
national  power:  the  economy.  Answering 
the  claim  by  Norman  D,  Levin  of  Rand 
for  “[a]  greater  link  between  U.S.  foreign 
policy  goals  and  domestic,  especially  eco¬ 
nomic,  objectives”  (1994,  pp.  xv-xvi), 

NSS  94  stresses 


"Top-level  political 
personalities  are 
more  likely  to  be 
reached  at  think 
tank  conferences  or 
symposia,  where 
they  deliver  keynote 
addresses  or  after- 
dinner  speeches,  and 
listen  to  selected 
panelists." 


such  economic 
objectives  as 
enhancing 
American  com- 
petitiveness, 
promoting  the 
partnership 
with  business 
and  labor,  and 
enhancing  ac¬ 
cess  to  foreign 


markets,  in  a 
way  that  leaves  little  doubt  about  the  will 


of  the  government  to  actively  invest  itself 
in  the  promotion  of  the  American  defense 
industry. 


How  Do  American  Think  Tanks  Work? 

One  could  deduce,  from  this  general 
survey  of  the  correlation  between  the  is¬ 
sues  discussed  by  think  tanks  in  their  pub¬ 
lications  and  the  content  of  NSS  94,  that 
these  organizations  create  and  inject  ideas 
in  the  policy  process  mainly  through  pam¬ 
phlets  or  books.  This  conclusion  would 
overestimate  the  influence  of  these  publi¬ 
cations.  These  documents  are  side-effects, 
if  not  residues,  of  the  real  value-added  ac¬ 
tivities  of  think  tanks  as  “ideas  brokers,” 
“brains  brokers,”  and  “personal  networks 
operators.”  Let  us  describe  briefly  these 
categories. 


The  Ideas  Brokers.  This  title  of  the 
very  well-informed  book  by  James  A. 
Smith  (1993)  captures  much  of  what  is  the 
purest  expression  of  their  activity. 
Whether  think  tankers  create  original  ideas 
(which  is  arguable),  or  more  likely  bring 
up  new  combinations  or  new  applications 
of  preexisting  ideas,  their  output  is 
basicaly  conceptual.  These  archetypes  of 
the  “symbolic  analysts”  (Reich,  1992)  de¬ 
liver  their  conceptual  product  to  numer¬ 
ous  customers:  first  of  all  to  the  senior  of¬ 
ficers,  civil  servants,  or  staffers  who  are 
the  concrete  actors  of  the  policy  process 
through  numerous  government  depart¬ 
ments,  bureaus,  committees,  or  agencies. 
Then  their  ideas  are  purveyed  to  the  top 
ranking  political  personnel  of  the  govern¬ 
ment:  secretaries  or  under-secretaries, 
chairpersons  of  diverse  committees,  and 
other  special  advisors.  And  finally  they 
submit  their  product  to  public  opinion,  or 
at  least  to  the  informed  part  of  it  that  con¬ 
stitutes  the  defense  community. 

Many  channels  are  used  for  this  activ¬ 
ity,  and  are  adapted  to  each  objective.  Top- 
level  political  personalities  are  more  likely 
to  be  reached  at  think  tank  conferences  or 
symposia,  where  they  deliver  keynote  ad¬ 
dresses  or  after-dinner  speeches,  and  lis¬ 
ten  to  selected  panelists.  Organizing  such 
events  is  therefore  a  large — and  often  lu¬ 
crative — part  of  think  tank  activity.  The 
intermediate  level  of  senior  actors  is  of¬ 
ten  treated  more  in  depth:  focus  work¬ 
shops  and  seminars  that  gather  restricted 
caucuses  allow  think  tankers  to  interact 
with  them,  to  trade  ideas  and  propose 
views.  This  private  interaction  is  often 
supplemented  by  more  public  and  official 
events,  such  as  testifying  before  Congres¬ 
sional  committees.  The  opportunity  for 
such  direct  interaction  naturally  depends 
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on  the  degree  of  proximity  or  of  political 
sympathy  between  the  think  tank  and  the 
current  administration.  This  explains  why, 
for  example,  DoD’s  analysts  from  the  In¬ 
stitute  for  National  Security  Studies 
(ENSS)  or  people  from  the  Brookings  In¬ 
stitution  may  have  been  more  active  than 
others  during  the  Democratic  adminis¬ 
tration’s  preparation  of  NSS  94, 

This  is  not  to  say  that  opposition  think 
tanks  have  remained  inactive:  the  Heri¬ 
tage  Foundation,  for  example,  has  been 
very  active  in  proposing  its  own  views  of 
the  national  security  strategy  (Holmes, 
1994)  to  such  powers  as  Rep.  Newt 
Gingrich,  Speaker  of  the  House  of  Repre¬ 
sentatives.  This  approach  has  included 
proposition  of  fully  crafted  pieces  of  leg¬ 
islation,  such  as  a  pending  bill  titled  the 
“National  Security  Act  of  1995.” 

All  these  private  meetings,  seminars,  or 
conferences  give  birth  to  another  impor¬ 
tant  activity:  publication.  Articles,  books, 
compilations  of  conference  presentations, 
etc.,  are  a  significant  part  of  think  tank 
activity,  and  of  their  revenue  (up  to  50%). 
But  they  are  not  so  much  the  channel 
through  which  think  tanks  operate,  as  they 
are  the  signs  by  which  the  outside  observer 
may  track  their  direct  interactive  activity. 

The  Brains  Brokers.  Think  tanks  do 
not  exist  on  ideas  alone.  They  also  require 
brains,  usually  associated  with  individu¬ 
als.  Staff  members  often  are  migrants  from 
senior  political  analyst  positions  or  derive 
from  diverse  kinds  of  fellowship  associa¬ 
tion  in  think  tanks;  some  staff  members 
come  from  senior  political  positions 
within  the  administration  or  the  Congress, 
and  vice  versa.  This  provides  a  privileged 
way  for  think  tanks  to  influence  the 
policy-making  process.  The  “revolving 


door,”  as  this  practice  has  come  to  be 
called,  has  been  used  intensely.  Pr.  Edward 
Warner  came  from  Rand  to  head  the 
Bottom-Up  Review  process  at  the  Penta¬ 
gon  as  Assistant  Secretary  for  Strategy  and 
Requirements.  Walter  Slocombe,  pres¬ 
ently  Under  Secretary  for  Policy,  came 
from  Brookings.  Secretary  of  Defense 
William  Perry  himself  elaborated  his  con¬ 
cept  of  cooperative  security,  in  associa¬ 
tion  with  John  D.  Steinburger  and  Ash 
Carter  (Assistant  Secretary  for  Interna¬ 
tional  and  Security  Policy),  as  a  distin¬ 
guished  fellow  at  Brookings. 

The  American  Enterprise  Institute  hosts 
such  distinguished  fellows  as  former  De¬ 
fense  Secretary  Dick  Cheney  or  United 
Nation  Ambassador  Jeane  Kirkpatrick, 
while  the  Center  for  Strategic  and  Inter¬ 
national  Studies  (CSIS)  has  welcomed 
such  individuals  as  Henry  Kissinger, 
Zbigniew  Brzezinski,  both  National  Se¬ 
curity  Advisers  and  a  former  Secretary  of 
State,  and  James  Schlesinger,  who  headed 
two  Cabinet  departments.  And  it  is  likely 
that  the  new  Republican  104th  Congress 
will  give  the  Heritage  Foundation  the  op¬ 
portunity  to  see  some  of  its  analysts  play 
a  direct  role  on  Capitol  Hill,  where  they 
will  be  able  to 

implement  the  "But  think  tanks  are 
policies  that  the  places  vrhere 
they  had  been  such  individuals  find 
designing  for  the  opportunity  to 


years  through  lormulate^  elabo- 

their  “Mandate  conlront,  en- 

forLeadership 

o  finally  diffuse  their 

(Butler,  Sanera, 

&  Weinrod,  '"f  * 

1984). 

It  would  be  inaccurate  to  presume  that 
these  individuals  and  their  ideas  are  mere 
products  of  think  tanks.  But  think  tanks 
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are  the  places  where  such  individuals  find 
the  opportunity  to  formulate,  elaborate, 
confront,  enrich,  validate,  and  finally  dif¬ 
fuse  their  ideas  through  their  collabora- 


I  would  argue 
fhaf  this  disconnec¬ 
tion  between  the 
administration  and 
the  think  tanks 
network  may  lead 
to,  or  be  the  sign  of, 
an  impoverishment 
of  the  relationship 
between  the  current 
government  and  the 
idea- creative  layers 
of  the  society/' 


tion  with  the 
regular  scholars 
that  constitute 
the  permanent 
core  of  the  orga¬ 
nizations,  Think 
tanks  provide 
them  with  the 
opportunity  to 
reach  a  selected 
audience.  And 
former  top- 
level  officials 
bring  to  think 
tanks  that  host 


them  some  priceless  aura  that  ensures  the 
success  of  the  events  they  support.  Like  a 
symbiosis,  this  often-enduring  relation¬ 
ship  between  these  individuals  and  think 
tanks  benefits  both  parties. 


Think  Tanks  as  Personal  Network 
Operators.  Beyond  their  functions  as  idea 
and  brains  brokers,  think  tanks  possess  one 
further,  and  fundamental,  feature.  As  they 
organize  private  meetings,  restricted  semi¬ 
nars,  and  public  symposiums,  and  trade 
senior  individuals  with  the  administration, 
think  tanks  build  and  operate  networks. 
They  link  different  worlds:  the  govern¬ 
ment  world,  the  business  world,  the  aca¬ 
demic  world,  and  the  military  world, 
whose  preoccupations  and  even  languages 
differ  so  much  from  each  other.  But  ideas 
do  not  circulate  by  themselves.  Think 
tanks  provide  the  venues — ^physical  or  vir¬ 
tual^ — where  people  periodically  gather 
and  interact.  I  would  suggest  that  it  is  for 
and  from  these  interactions  that  think  tanks 


build  the  networks  that  constitute  the  very 
base  of  the  policy-making  system.  This 
activity  passes  largely  unnoticed  and  is 
nearly  impossible  to  trace. 


What  Was  the  Overall 
Contribution  to  NSS  94? 

Curiously  enough,  all  the  actors  in  the 
process  that  were  interviewed  said  that 
NSS  94  received  very  few  direct  contri¬ 
butions  from  think  tanks.  However,  we 
have  seen  that  the  issues  themselves  that 
were  addressed  in  the  document  have  been 
comprehensively  discussed  and  published 
by  several  think  tanks.  It  seems  therefore 
that  all  this  activity  has  taken  place  either 
beside  or  much  ahead  of  the  formulation 
of  the  document  itself.  The  chaotic  pro¬ 
cess  that  we  described  earlier  may  be  re¬ 
sponsible  in  part  for  that  shortcoming:  it 
seems  that  the  administration  was  too  busy 
struggling  with  its  own  internal  frictions 
to  pay  much  attention  to  the  outside  world. 
Should  we  view  this  as  the  result  of  ex¬ 
cessive  self-confidence  on  the  part  of  the 
administration?  Or  on  the  contrary,  did 
policy  makers  consider  this  writing  as  a 
futile  and  formal  exercise  that  had  to  be 
achieved  at  the  least  intellectual  cost  in 
order  to  concentrate  on  more  short-term 
but  really  burning  issues,  such  the  chang¬ 
ing  relationships  with  Russia  or  China,  or 
the  Bosnia  crisis? 

Whatever  the  cause,  I  would  argue  that 
this  disconnection  between  the  adminis¬ 
tration  and  the  think  tank  network  may 
lead  to,  or  be  the  sign  of,  an  impoverish¬ 
ment  of  the  relationship  between  the  cur¬ 
rent  government  and  the  ideas-creative 
layers  of  the  society.  We  must  look  to  the 
Congress  and  the  Clinton  Administration 
to  see  whether  this  relationship  is  revital- 
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ized  and  to  stress  the  need  for  a  renewed 
and  enhanced  political  debate.  In  any  case, 
every  think  tank  analyst  could  endorse  this 
claim  by  Rand’s  Ronald  D.  Asmus  (1994, 
p.  ix):  “If  the  United  States  is  to  find  a 
new  post-Cold  War  consensus,  then  air¬ 
ing  and  debating  these  views  and  differ¬ 
ences  is  a  healthy  and  inevitable  part  of 
building  this  new  consensus.” 

Think  Tanks  ih  the  Frehch  Process 
Who  Are  the  French  Tanks? 

Some  basic  distinctions  exist  between 
French  think  tanks  and  their  American 
counterparts.  We  may  define  them  in  vari¬ 
ous  ways.  The  more  efficient  is  to  come 
back  to  Smith’s  definition  (1993,  p,  xiii): 
“the  private,  nonprofit  research  groups  that 
operate  on  the  margins  of  the  nation’s  for¬ 
mal  political  processes.”  On  each  point  of 
this  definition,  we  can  find  a  wide  range 
of  different  patterns  for  these  think  tanks 
in  France,  as  well  as  in  the  United  States. 

Let  us  consider  first  the  status  of  the 
group.  We  shall  find  groups  ranging  from 
the  totally  private  AERO  company,  and 
“Loi  de  1901”  nonprofit  associations  such 
as  the  Center  for  Strategic  Research 
(CREST)"^  or  foundations  such  as  the  re¬ 
cent  Fondation  pour  les  Etudes  de 
Defense,  to  government  in-house  groups 
such  as  the  Defense  Acquisition  Agency’s 
(DGA)  Center  for  Defense  Analysis,  or  the 
Delegation  aux  Affaires  Strategiques 
(DAS), 

As  in  the  United  States,  a  number  of 
think  tanks  are  university-linked;  but  those 
are  generally  very  limited  teams  (and 
sometimes  single  individuals).  Except  per¬ 
haps  for  Pr.  Schmitt’s  studies  in  the  de¬ 
fense  economy  field  at  Paris-Porte  Dau- 


phine  University,  these  teams  do  not  usu¬ 
ally  make  a  significant  contribution. 

The  Institute  for  National  Security 
Studies,  hosted  here  by  the  National  De¬ 
fense  University,  has  a  near  equivalent  in 
the  Groupes  d’ Etude  et  de  Recherche 
hosted  by  the  Center  of  Higher  Armament 
Studies  in  Paris. 

Some  think  tanks  are  more  closely 
linked  to  the  political  world:  various  foun¬ 
dations,  “Centre  d’ Etudes,”  forums  or 
“Carrefours  de  Reflexion”  act  as  idea  pro¬ 
viders,  diffusers,  or  catalysts  for  groups 
or  individuals  pursuing  some  political  ob¬ 
jectives.  In  some  cases,  they  seem  to  be 
more  like  temporary  refuges  where 
out-of-office  senior  political  personal  find 
an  active  rest  before  a  hypothetical  return 
to  power. 

What  are  the  sources  of  funding  for 
these  groups?  Here  again,  the  answers 
vary.  Many  de- 

pend  on  govern-  "French  tradition 
ment  contracts  dees  not  encourage 
for  the  major  such  private  interest 
part  of  their  ac-  (endovmients  and 
tivities.  Here,  private  funding)  in 
they  clearly  di^  ncrtional  security 
ferfromAmeri-  thinking." 
can  think  tanks, 

many  of  which  rely  mainly  on  endow¬ 
ments  and  private  funding,  or  are  partly 
self-funded  through  profitable  publica¬ 
tions  and  conferences.  French  tradition 
does  not  encourage  such  private  interest 
in  national  security  thinking. 

The  Objectives  of  French  Think  Tanks 

Do  French  think  tanks  mainly  pursue 
only  educational  and  research  objectives, 
or  do  they  also  try  to  influence  the 
policymaking  process?  Here  also,  like 
American  think  tanks,  their  aims  vary: 
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some  sponsor  pure  thinkers  (if  there  are 
any  in  this  world) — ^the  Woodrow  Wilson 
Scholars  nearly  mirror  the  Centre  National 
de  la  Recherche  Scientifique  researchers 
in  foreign  and  defense  policy;  others  are 
openly  partisan,  such  as  the  left-wing  In¬ 
stitute  for  International  Relations  and 
Strategy  (IRIS)  and  the  right-wing  Cen¬ 
ter  for  Prospective  and  Strategic  Studies 
(CEPS).  We  must  keep  in  mind  that  the 
French  Parliament  is,  traditionally  if  not 
constitutionally,  much  less  involved,  or  at 
least  at  a  lesser  detailed  level,  in  foreign 
policy  and  national  security  policy  than 
its  American  counterpart.  There  is  a  strong 

tradition  under 
the  institutions 
of  the  French 
Fifth  Republic 
to  recognize  the 
very  preemi¬ 
nent  preroga¬ 
tive  of  the  presi¬ 
dent  in  this  area. 
As  a  result,  the 
French  system 
allows  the  executive  bureaucracy  to  shape 
these  policies  without  so  much  interfer¬ 
ence  from  either  the  political  parties,  or 
the  Parliament.  The  need  for  circulating 
ideas  is  therefore  less  important  and,  con¬ 
sequently,  the  input  from  French  think 
tanks  much  less  visible  in  the  policy  pro¬ 
cess. 


need  for  circu- 
lating  ideas  is  there 
fore  less  important 
and,  consequently, 
the  input  from 
French  think  tanks 
much  less  visible  in 
the  policy  process." 


Fewer  think  tanks,  smaller  in  size  and 
with  fewer  financial  resources,  are  less  in¬ 
fluential  in  the  policy-making  process — 
this  is  how  French  think  tanks  compare  to 
U.S.  ones,  which  have  been  quite  prop¬ 
erly  pictured  as  a  “quintessentially  Ameri¬ 
can”  institution  (Smith,  1993,  p.  xiii). 


Did  French  Think  Tanks 
Contribute  to  the  White  Paper? 

There  is  no  need  here  to  look  in  great 
detail  at  how  French  think  tanks  usually 
operate:  by  and  large,  they  use  the  same 
means  as  their  American  counterparts. 
Seminars  or  conferences,  testifying  before 
government  or  Parliamentary  commis¬ 
sions,  publications — French  think  tanks 
do  all  these,  although  on  a  lesser  scale, 
because  of  their  lower  visibility  in  the  po¬ 
litical  landscape,  and  the  relative  lack  of 
interest  shown  them  by  the  administration. 

Think  tanks  published  several  books  or 
articles  during  the  preparation  of  the  white 
paper.  For  instance,  in  a  special  issue  of 
the  Defense  Nationale  journal,  chairman 
of  Comite  d’ Etudes  de  Defense  Nationale 
Paul-Marie  de  la  Gorce  suggested  a  broad 
national  security  strategy  framework.  He 
emphasized  the  need  of  defining  first  the 
national  interests:  “[The  threats]  can  only 
be  defined  relative  to  those  interests  that 
have  to  be  defended,  and  one  should  be¬ 
gin  by  defining  these  latter”  (1993,  p.  10). 
His  articulation  of  the  geo-strategic  con¬ 
text  relied  on  four  premises:  (a)  the  exist¬ 
ence  of  one  military  superpower;  (b)  the 
existence  of  several  nuclear  powers;  (c) 
the  new  instability  of  the  Eurasian  conti¬ 
nent,  from  the  Adriatic  sea  to  Chinese 
borders;  and  (d)  the  growing  instability  of 
those  “strategic  zones  where  political,  and 
eventually  military,  instability  comes  from 
economic  and  social  crises,  with  their  eth¬ 
nical  and  religious  spillovers”  (la  Gorce, 
1993,  p.  15). 

From  these  premises,  de  la  Gorce  de¬ 
rived  several  propositions,  specially  ad¬ 
vocating  for  nuclear  deterrence  and  con¬ 
ventional  force  projection  capacities. 
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In  the  same  issue,  former  chairman  of 
the  Fondation  pour  les  foudes  de  Defense 
Nationale,  Pierre  Dabezies,  published 
“Reflexions  sur  le  Livre  Blanc”  (1993). 
His  main  argument  stressed  the  need  to 
substitute  an  “approach  by  the  political 
project”  for  a  mere  static  assessment  of 
national  interests.  From  that  perspective, 
he  discussed  how  cross  choices  between 
different — and  maybe  contradictory — 
projects  muddle  the  French  strategic  land¬ 
scape:  the  pursuit  of  the  global  “new  world 
order,”  the  tightening  of  links  with  NATO, 
the  building  of  the  European  Union  and 
of  a  hypothetical’  foreign  and  security 
common  policy”  being  among  the  most 
burning  issues. 

These  discussions  were  clearly  in  the 
scope  of  the  white  paper.  Nevertheless, 
they  were  more  broadly  political, 
editoriallike  assessments  than  scholarly 
studies  usually  expected  from  think  tanks. 
Further,  there  was  no  indication  that  these 
pieces  reflected  anything  more  than  per¬ 
sonal  and  selective  views — not  being  part 
of  more  broad  research  within  any  orga¬ 
nized  framework. 

An  exception  is  the  book  published  by 
CREST  director  Alain  Baer  titled: 
Thoughts  on  the  Nature  of  Future  Defense 
Systems  (1993).  This  book  originated  in 
the  work  of  one  of  the  research  groups  that 
operate  in  connection  with  the  Center  for 
Higher  Armaments  Studies. It  presents  a 
comprehensive  approach  of  new  defense 
concepts,  both  in  terms  of  organization 
and  material,  especially  stressing  the  im¬ 
portance  of  real-time  intelligence.  It  pro¬ 
posed  creation  of  a  permanent  National 
Security  Council,  on  the  American  model, 
and  a  new  concept  for  the  organization  of 
the  forces,  which  would  be  articulated  be¬ 
tween  interior  and  exterior  forces,  accord¬ 


ing  to  whether  their  vocation  would  be  to 
defend  the  metropolitan  territory,  or  to  be 
projected  abroad.  Alain  Baer  also  provided 
a  fairly  extensive  study  of  the  current 
“military  revolution”  and  of  its  implica¬ 
tions  in  terms  of  technology  and  weapons 
systems. 

This  study  was  clearly  original  among 
other  more  political  and  circumstantial 
publications,  and  was  probably  the  only 
one  that  could  be  considered  as  a  think 
tank  product. 


What  Has  Beeh  the  Real  Ihput  from 
Think  Tanks  to  the  White  Paper? 

Specifically  in  the  case  of  the  white  pa¬ 
per,  it  appears  from  the  interviews  com¬ 
pleted  that  the 

ad  hoc  commis-  book]  presents 

sion  had  very  ^  comprehensive 
little  input  from  approach  of  new 
think  tanks:  (a)  defense  concepts. 

Just  one  inde-  both  in  terms  of 


pendent  expert  or gonixation  and 

in  the  commis-  especially 

Sion  itself  was  »»*« 

the  head  of  a  !""‘*.** 

,  .  ,  ,  , ,  inteiligence." 

think  tank  (the  _ 

French  Institute 

for  International  Relations);  (b)  no  request 
for  any  advice  was  ever  made  by  the  com¬ 
mission  to  think  tanks  themselves,  al¬ 
though  the  commission  heard  many  ex¬ 
perts  intuitu  personae',  and  (c)  the  com¬ 
mission  received  several  spontaneous  con¬ 
tributions  from  diverse  organizations,  such 
as  professional  syndicates  or  corporative 
groupments,  but  none  of  them  standing 
think  tanks. 

As  in  the  case  of  the  writing  of  NSS 
94,  some  external  factors  may  have  con¬ 
tributed  to  keeping  think  tanks  apart  from 
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the  policy  process.  For  example,  it  has 
been  largely  noticed  that  this  white  paper 
was  a  product  of  the  so-called  “cohabita¬ 
tion,”  which  is  a  strange  situation  of  gov¬ 
ernment  in  which  a  left-wing  president 

shares  the 
power  with  a 
right- wing 
prime  minister. 
This  kind  of  po¬ 
litical  compro¬ 
mise  seems  to 
have  become  a 
French  “favor¬ 
ite”  for  nearly 
10  years.  If  it 
has  led  to  some 


^^[The  French  admin- 
isf ration]  therefore 
has  less  inclination 
than  the  American 
one  to  look  outside 
for  ideas  or  concep¬ 
tual  achievements 
that  it  presumably 
can  provide  on  its 
own/' 


serious  con¬ 
frontation  on  domestic  issues,  it  has  never 
seriously  affected  the  unity  of  action  of 
the  government  on  foreign  policy  and  de¬ 
fense  issues.  But,  it  may  have  influenced 
the  autonomy  of  thought  of  the  commis¬ 
sion  and  its  ability  to  launch  into  more  cre¬ 
ative  strategic  thinking.  The  writers  of  the 
white  paper  could  therefore  have  per¬ 
ceived  their  assignment  as  not  so  much  to 
bring  up  new  ideas  (which  could  have  jus¬ 
tified  resorting  to  think  tanks),  but  to 
achieve  a  balanced  and  smooth  product 
that  could  get  the  broadest  political  assent 
possible. 


Comparison  of  the  TWo  Paherns 


In  both  France  and  the  United  States, 
organizations  that  more  or  less  fit  the  de¬ 
scription  of  a  think  tank  usually  operate 
at  the  margin  of  the  national  security 
strategy-making  process.  These  organiza¬ 
tions  are  much  more  common  in  the 
United  States  and  enjoy  a  higher  visibil¬ 


ity,  because  of  their  number,  size,  and  tra¬ 
ditional  implication  in  political  life. 

Specifically  concerning  NSS  94  and  the 
French  white  paper,  think  tanks  seem  to 
have  had  little  direct  input.  In  the  United 
States,  where  they  have  nevertheless  been 
very  active  in  debating  the  major  issues 
related  to  the  national  security  strategy, 
they  probably  suffered  from  a  lack  of  in¬ 
terest  by  the  administration  for  their  in¬ 
put.  But  this  current  loose  relationship  be¬ 
tween  U.S.  think  tanks  and  the  adminis¬ 
tration  is  unusual.  It  is,  however,  the  rule 
in  France,  where  these  institutions  are  tra¬ 
ditionally  kept  outside  the  political  pro¬ 
cess  by  an  administration  more  confident 
in  its  own  capacity  to  elaborate  the  policy. 
Among  other  reasons,  this  major  differ¬ 
ence  between  the  two  countries  may  be 
related  to  the  history  of  their  form  of  gov¬ 
ernment.  This  observation  by  Smith  seems 
true  today:  “...  in  countries  with  older  civil 
service  tradition  and  fewer  political  ap¬ 
pointments,  experts  could  be  found  in  the 
bureaucracy  much  earlier  than  they  could 
in  the  American  system,  where  nonparti¬ 
san  experts  typically  had  to  be  housed  on 
the  outside”  (1993,  pp.  228-229).  From 
that  perspective,  the  long  American  tradi¬ 
tion  of  distrust  for  a  centralized  govern¬ 
ment  and  the  chronic  instability  of  the 
upper  layers  of  the  administration  (be¬ 
cause  of  the  great  number  of  short-lived 
political  appointees),  have  probably  con¬ 
tributed  to  enhance  and  sustain  the  impor¬ 
tance  of  think  tanks  in  political  life.  These 
organizations  assume  a  central  function  in 
improving  the  continuity  of  thought  of  the 
nation,  and  in  helping  elaborate  political 
wisdom.  Yet,  if  we  agree  with  Smith’s 
statement  (1993,  p.  xi)  that  “[tjhere  is 
something  troubling  about  the  relationship 
among  experts,  leaders,  and  citizens  that 
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tends  to  make  American  politics  more 
polarized,  shortsighted,  and  fragmented — 
and  often  less  intelligent — than  it  should 
be,”^  we  may  assume  that  even  think  tanks 
fall  short  of  the  task. 

On  the  French  side,  think  tanks  are 
granted  only  a  little  influence  in  policy 
making.  We  have  already  stressed  the 
major  feature  of  the  French  administra¬ 
tion  as  a  very  intellectually  autonomous 
one:  several  centuries  of  government  ser¬ 
vice,  a  high  ideal  of  public  service,  a  long 
history  of  centralized  power  from  Colbert 
up  to  the  present,  make  this  administra¬ 
tion  think  of  itself  as  the  one  of  the  best 
and  the  brightest.  It  therefore  has  less  in¬ 
clination  than  the  American  one  to  look 
outside  for  ideas  or  conceptual  achieve¬ 
ments  that  it  presumably  can  provide  on 
its  own. 

We  have  described  American  think 
tanks  as  key  actors  in  building  the  personal 
networks  that  vitalize  the  policy-making 
process.  In  France  also,  personal  networks 
are  fundamental  in  the  policy  process;  the 
difference  is  that  these  networks,  in  a 
sense,  preexist  through  the  links  that  gath¬ 
ered  many  senior  military  or  civil  servants 
from  their  education  in  the  two  or  three 
major  institutions  that  form  the  French 
elite  (Alumni  of  the  Ecole  Nationale 
d’ Administration  or  of  the  Ecole 
Polytechnique,  for  example,  operate  very 
far-reaching  and  powerful  networks  that 
cross  the  highest  layers  of  the  political, 
administrative,  and  business  worlds.) 
Therefore  less  room  exists  in  that  func¬ 
tion  for  organizations  such  as  think  tanks. 
I  would  suggest  here  that  this  pattern  of 
policy  making  will  soon  find  its  own  lim¬ 
its:  As  the  European  integration  process 
goes  on,  these  education-based  national 
networks  will  no  longer  operate  at  a  suffi¬ 


cient  level.  Since  the  “Euro-Elite”  will 
obviously  be  more  diversified  than  the 
national  French  one,  there  certainly  will 
be  a  need  for  new  links  and  venues  where 
the  “Eurocrats,”  as  they  are  sometimes 
called  (a  bit  pejoratively),  will  be  able  to 
interact  and  forge  together  their  political 
thought. 

Conclusion 


As  David  M.  Ricci  states  (1993,  p.  182): 
“Think  tanks  fit  somewhere  into  public 
life,  but  no  one  knows  exactly  where  that 
is,” 

This  essay  has  tried  to  investigate  how 
they  fit  into  the  national  security  strategy 
formulation  process.  How  they  operate  in 
this  very  specific  field  in  the  United  States, 
compared  with  the  French  way,  suggests 
that  these  patterns  do  not  only  depend  on 
internal  features 

of  these  organi-  //this  essay  has  tried 
zations,  but  also  to  investigate  how 
reflect  major  [think  tanks]  fit  into 
traits  of  the  the  national  secori^ 
policy-making  strategy  formulation 
process.  The  process." 
importance  of 

their  input  in  the  U.S.  process,  their  vis¬ 
ibility  in  political  life,  and  their  very  ex¬ 
istence  are  fundamentally  linked  to  the 
characteristics  of  the  American  political 
system.  This  is  confirmed  by  contrast  with 
the  French  example,  and  this  observation 
agrees  with  the  thesis  expressed  (from  a 
political  science  perspective)  by  Ricci 
(1993)  and  from  a  more  historical  one  by 
Smith(1993).  Smith  finds  think  tanks  a 
specific  solution  to  the  no-less-specific 
problem  of  the  American  policy-making 
system.  More  precisely,  they  help  to  build 
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and  operate  the  personal  networks  that  fa¬ 
cilitate  the  complex  interactions  between 
and  among  the  industrial,  business,  aca¬ 
demic,  and  political  worlds;  and  then  be¬ 
tween  these  civilian  entities  and  the  ad¬ 
ministration  and  the  Congress,  as  the  two 
major  protagonists. 

Finally,  I  would  suggest  that  this  hy¬ 
pothesis  could  be  confirmed  by  observ¬ 
ing  the  policy-making  system  of  the  Eu¬ 


ropean  Union:  In  the  same  way  that  think 
tanks  emerged  in  the  United  States  after 
World  War  II,  and  spectacularly  increased 
in  the  1970s  and  1980s,  accompanying  the 
increase  in  the  federal  government,*^  we 
should  expect  similar  organizations  to 
emerge  and  develop  at  the  periphery  of 
the  European  Commission  in  Brussels, 
following  the  implementation  of  the 
Maastricht  treaty. 
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Appendix  A  -  Extract  from  50  USC  404a 


Each  national  security  strategy  report 
shall  set  forth  the  national  security  strat¬ 
egy  of  the  United  States  and  shall  include 
a  comprehensive  description  and  discus¬ 
sion  of  the  following: 

(1)  The  worldwide  interests,  goals,  and 
objectives  of  the  United  States  that 
are  vital  to  the  national  security  of 
the  United  States. 

(2)  The  foreign  policy,  worldwide  com¬ 
mitments,  and  national  defense  ca¬ 
pabilities  of  the  United  States  nec¬ 
essary  to  deter  aggression  and  to 
implement  the  national  security  strat¬ 
egy  of  the  United  States. 


(3)  The  proposed  short-term  and 
long-term  uses  of  the  political,  eco¬ 
nomic,  military,  and  other  elements 
of  the  national  power  of  the  United 
States  to  protect  or  promote  the  in¬ 
terests  and  achieve  the  goals  and 
objectives  referred  to  in  paragraph 
(1). 

(4)  The  adequacy  of  the  capabilities  of 
the  United  States  to  carry  out  the  na¬ 
tional  security  strategy  of  the  United 
States,  including  an  evaluation  of  the 
balance  among  the  capabilities  of  all 
elements  of  the  national  power  of  the 
United  States  to  support  the  imple¬ 
mentation  of  the  national  security 
strategy. 
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Appendix  B  -  Structural  Comparison  of  the  Two  Basic  Documents 


A  National  Security  Strategy  of  Engage¬ 
ment  and  Enlargement 

Preface  (W.  J.  Clinton) 


I  Introduction 


II  Advancing  Our  Interests 


IL 1  Enhancing  Our  Security 

Maintaining  a  Strong  Defense  Capability 
Major  Regional  Contingencies 
Overseas  Presence 
Counterterrorism ... 

Combatting  Terrorism 
Fighting  Drug  Trafficking 
Other  Missions 


Livre  Blanc  sur  la  Engagement  Defense 
1994 

Preface  (E.  Bahadur) 

Preface  (F.  Leotard) 

Introduction 

First  Part:  The  Strategic  Context 

1 .  International  Prospect 

2.  Defense  Policy  Objectives 

2. 1  Defending  France’s  Interests 

2.2  Constructing  Europe 

2.3  Implementing  a  Global  Conception 

of  Defense 

3.  International  Reference  Framework 

3.1  Toward  a  New  European  Security 

3.2  Reinforcing  U.N.  Role 

3.3  Bilateral  Cooperation 

4.  Our  Defense  Strategy 

5 .  The  Capabilities  of  the  Armed  Forces 
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Deciding  When  and  How  to  Employ  U.S.  3.4  Arms  Control,  Disarmament  and 

Forces  Nonproliferation  Treaties 

Combatting  the  Spread  and  Use  of  Weap¬ 
ons  of  Mass  Destruction 

Nonproliferation  and  Counterproliferation 

id 

Strategic  Nuclear  Forces 
Arms  Control 
Peace  Operations 

Strong  Intelligence  Capabilities  9.5 

The  Environment 

6. 

2.2  Promoting  Prosperity  at  Home  9.4 

Enhancing  American  Competitiveness  7. 

Partnership  with  Business  and  Labor  7.3 

Enhancing  Access  to  Foreign  Markets  7.4 

The  NAFTA 

Asia  Pacific  Economic  Cooperation 
Uruguay  Round  of  GATT 
U.S. -Japan  Framework  Agreement 
Expending  the  Realm  of  Free  Trade 

Strengthening  Macroeconomic  Coordina¬ 
tion 

Providing  for  Energy  Security 

Promoting  Sustainable  Development 
Abroad 

Promoting  Democracy 


Organization  of  Information 

Human  Resources 

Economic  Defense 

Arms  Policy  and  Industrial  Strategy 

New  State/ Arms  Industry  Relation¬ 
ship 

An  Export  Policy 
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III  Integrated  Regional  Approches 

Europe  and  Eurasia 

East  Asian  and  the  Pacific 
The  Western  Hemisphere 
The  Middle  East,  Southwest,  and 
South  Asia 
Africa 

9.  Defense  and  Society 

IV  Conclusions  Conclusion 


20 


Think  Tanks  and  the  National  Security  Strategy  Formulation  Process 


Appendix  C  -  Composition  of  the  White  Paper  Commission 


Chairman 

Marceau  Long,  vice-chairman  of  the 

Conseil  d'Etat 

Office  of  the  Prime  Minister 

General  Schmitt,  special  advisor 

Prefet  Marland,  advisor  for  domestic 
affairs 

Rear  Admiral  Lecointre,  head  of 
military  staff 

General  Lerche,  head  of  Secretariat 
General  de  la  Defense  Nationale 

Defense  Ministery 

M,  Donnedieu  de  Vabres,  special 
assistant 

Admiral  Lanxade,  chief  of  the  joint 
staff 

General  Conze,  head  ofDGA  (De¬ 
fense  Acquisition  Agency) 

M.  Roussely,  general  secretary  for 
administration 

General  Rannou,  head  of  military 
staff 

M.  Mallet,  director  for  strategic 
affairs 

State  Department 

M.  Racine,  special  assistant 

M.  Guehenno,  head  of  CAP  (Center 
for  Analysis  and  Prospective 
Studies) 

M.  Barry-Delongchamps,  director  for 
strategic  affairs 


Ministry  of  Treasury 

Ms.  Bouillot,  budget  director 

Ministry  of  Interior 

Prefet  Riolacci 

Ministry  of  Research 

M.  Paolini,  special  assistant 

Ministry  of  Industry 

M.  Lombard,  director  for  industrial 
strategy 

Ministry  of  International  Cooperation 

M.  Pouillieute,  head  of  staff 

Atomic  Energy  Agency 

M.  Baleras,  director  for  military 
applications 

Independent  Experts 

M.  de  Montbrial 
M.  Levy 

M.  Prada,  Cour  des  Comptes  ( Gen¬ 
eral  Accounting  Office) 
Ambassador  Robin 
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Appendix  D  -  Interviews  Completed 


In  the  U.S.: 

Col.  Richard  Barry,  Institute  for 
National  Security  Studies 

Bruce  Blair,  The  Brookings  Institution 

Lawrence  DiRita,  The  Heritage 
Foundation 

Capt.  Keith  Hans,  National  Security 
Council  Staff 

Patrick  Glynn,  American  Enterprise 
Institute 

Robert  Grant,  director  U.S.  CREST 

Eric  Peterson,  vice-president,  Center 
for  Strategic  and  International 
Studies 


In  France: 

Col.  (Armt.  Corp)  Patrick  Auroy, 

Livre  Blanc  Commission  Secretariat 

General  (Armt.  Corp)  Alain  Cremieux, 
former  Commandant  of  the  Center 
for  Higher  Armament  Studies 
(CHEAr) 

Jean-Francois  Delpech,  director  of  the 
Center  for  Strategic  Research 
(CREST) 

General  Eric  de  la  Maisonneuve, 
director  of  the  Foundation  for 
Defense  Studies  (FED) 

Thierry  de  Montbrial,  director  of  the 
French  Institute  for  International 
Relations  (IFRI) 
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End  Notes 


1 .  In  French  government  institutions,  the 
“Conseil  d’Etat”  is  an  equivalent  of  the 
U.S.  Supreme  Court,  in  its  role  of  judg¬ 
ing  the  disputes  arising  between  the  in¬ 
dividuals  and  the  state.  It  also  enjoys 
substantial  legislative  and  regulatory 
powers. 

2.  This  structure  has  a  function  and  a 
composition  similar  to  those  of  the  U.S. 
National  Security  Council.  But  unlike 
the  NSC,  it  has  no  substantial  support¬ 
ing  bureaucracy. 

3.  The  Heritage  Foundation  operates  a  24- 
hour  online  computer  service  having 
an  explicit  name:  ‘Town  Hall — The 
Conservative  Meeting  Place.” 

4.  It  may  be  worth  noting  that  CREST  has 
the  rather  exceptional  feature  of  hav¬ 
ing  a  U.S. -based  sister  organization, 
US  CREST,  in  Arlington,  VA,  whose 
vocation  is  specifically  to  work  on 
transatlantic  issues. 


5.  This  institution  is,  although  on  a  lesser 
scale,  an  equivalent  of  the  Industrial 
College  of  the  Armed  Forces. 

6.  The  Research  Institute  of  the  Western 
European  Union  headed  by  John  Roper 
in  Paris  may  represent  a  prototype  (al¬ 
though  certainly  specific  in  its  belong¬ 
ing  to  the  WEU)  of  these  future  Euro¬ 
think  tanks. 

7.  The  increase  in  the  number  of  White 
House  staff  is  a  good  indicator  of  this 
rise  of  “big  government”:  the  staff 
comprised  just  48  people  in  1944  un¬ 
der  President  Franklin  D.  Roosevelt.  It 
rose  to  275  in  1960  under  President 
Dwight  D.  Eisenhower,  and  increased 
to  540  in  1975  under  President  Gerald 
Ford.  Today,  the  Executive  Officer  of 
the  President  consists  of  about  2,000 
individuals. 
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RCTHINKING  TWENTY-FIRST 
CENTURY  ACQUISITION: 
EMERGING  TRENDS  FOR 
EFFICIENCY  ENDS 

Conrads,  Ckcotello,  Steve  G.  Green,  and  Martin  1,  Hornyak 


Understanding  three  emerging  public-sector  trends—privatization,  promotion 
of  competition,  and  continuous  process  improvement — is  essential  to  achieve 
efficiencies  sought  by  acquisition  managers  in  the  21st  century. 


With  the  collapse  of  the  Soviet 
threat  and  a  steady  decline  of 
available  resources,  the  Depart¬ 
ment  of  Defense  (DoD)  is  adapting  to  un¬ 
precedented  change.  The  military  services 
in  the  1990s  have  introduced  profound 
changes  to  DoD’s  financial  and  resource 
management  systems,  compared  to  when 
former  Secretary  of  Defense  Robert 
McNamara  first  introduced  the  Planning, 
Programming,  and  Budgeting  System 
(PPBS)  (Hough,  1992).  The  Defense  Per¬ 
formance  Review  (DPR),  Chief  Financial 
Officer’s  Act  (CFO),  and  Government  Per¬ 
formance  and  Results  Act  (GPRA)  all  gen¬ 
erate  policy  and  legislation  demanding 
considerations  and  measurement  of  effi¬ 
ciency  in  DoD  operations.  Public  func¬ 
tions  that  fail  to  meet  the  call  for  improved 
efficiency  are  being  privatized  or  consoli¬ 
dated  away. 


Acquisition  management  is  not  immune 
from  these  demands.  Program  managers 
must  understand  and  take  advantage  of 
these  new  initiatives  to  improve  efficiency 
or  face  possible  elimination  as  public  em¬ 
ployees.  Here  we  examine  three  current 
public-sector  management  policy 
trends:  (a)  privatization  of  functions,  (b) 
promotion  of  competition  between  gov¬ 
ernment  entities  or  civilian  contractors, 
and  (c)  continuous  process  improve¬ 
ment.  For  program  managers,  determin¬ 
ing  which  policies  to  consider  and 
implement  will  be  a  daunting  challenge. 
To  be  successful,  an  acquisition  man¬ 
ager  must  be  aware  of  policy  initiatives 
and  develop  means  of  assessing  which 
policy  options  are  best.  Here  we  outline 
some  current  policy  initiatives  affecting 
the  acquisition  community,  then  inves¬ 
tigate  several  assessment  alternatives 
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that  make  use  of  applicable  private-sec¬ 
tor  business  metrics  for  policy  evalua¬ 
tions. 

The  Environment 


The  American  defense  budget  has  been 
shrinking  in  real  terms  since  the  mid- 
1980s.  From  1985  through  1997,  DoD  will 
experience  overall  defense  budget  cuts  to 
equal  42%  (including  a  30%  reduction  in 
personnel),  overseas  base  closures  of  35%, 
and  U.S.  base  shutdown  of  15%  (DoD 
Comptroller,  1994).  Despite  these  cuts,  the 
requirements  to  support  critical  national 
interests  in  various  regions  of  the  world 
remain.  With  budgets  shrinking  and  re¬ 
quirements  steadily  growing,  DoD  has 
logically  focused  on  initiatives  to  increase 
efficiency.  Acquisition  management  has 
seen  a  host  of  formal  acquisition  improve¬ 
ment,  streamlining,  and  reform  initiatives 
that  suggest  or  direct  how  to  improve  the 
way  in  which  weapon  systems  are  pro¬ 
cured  (Harman,  1995).  Most  recently  the 
Federal  Acquisition  Streamlining  Act  of 
1994  culminated  a  concerted  effort  to 
place  reform  of  government  procurement 
at  the  top  of  the  list  of  national  priorities 
(van  Opstal,  1995). 

Despite  these  reform  attempts,  numer¬ 
ous  factors — such  as  DoD’s  complex  or¬ 
ganizational  structure,  lack  of  incentives 


to  improve  operations,  dynamic  require¬ 
ments,  and  shifting  direction — hamper 
efforts  to  improve  acquisition  manage¬ 
ment.  A  recent  study  requested  by  the  Of¬ 
fice  of  the  Under  Secretary  of  Defense  for 
Systems  Integration  investigates  the  con¬ 
flicts  of  roles  and  incentives  inherent  in 
acquisition  management  (Fox,  Hirsch,  & 
Krikorian,  1995).  The  study  highlights 
critical  issues  within  the  defense  acquisi¬ 
tion  culture,  and  suggests  that  acquisition 
program  managers  top  priority  is  keeping 
their  programs  alive  and  moving  through 
the  acquisition  process.  This  can  occur 
even  when  the  program’s  completion  may 
not  be  consistent  with  larger  interests.  A 
1992  General  Accounting  Office  (GAO) 
study  concludes  that  traditional  targets  of 
acquisition  reform  including  performance 
shortfalls,  schedule  delays,  and  cost  in¬ 
creases,  are  the  logical  consequences  of 
the  current  acquisition  culture  (GAO/ 
NSIAD-93-15, 1992). 

Public  sector  organizations  often  have 
problems  achieving  efficiency  without 
compromising  mission.  Prager  (1994)  ar¬ 
gues  the  public  sector  is  inefficient  be¬ 
cause  of  “...a  lack  of  political  will  to  es¬ 
tablish  efficiency  as  a  high-level  priority 
of  government  operations,”  He  argues  that 
public  sector  management  is  not  given 
with  sufficient  flexibility  to  pursue  effi¬ 
ciency  goals.  Improving  public  sector  ef¬ 
ficiency  is  further  complicated  by  an  in- 
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centive  structure  that  is  neutral  toward  or 
even  discourages  cost  savings  (Prager, 
1994).  In  addition,  Kelman  (1990)  sug¬ 
gests  that  the  lack  of  competition,  autono¬ 
mous  choice,  profit  motivation,  and  timely 
procurement  source  selection  decisions 
undermine  government  efficiency.  Values 
such  as  equitable  distribution  of  govern¬ 
ment  contracts  for  procured  goods  may 
also  compete  with  efficiency  goals.  Fi¬ 
nally,  Meier  (1987)  argues  that  efficiency 
within  many  public  organizations  is  not 
even  a  relevant  goal  and  “the  goal  of  gov¬ 
ernment  agencies  is  universal  service 
rather  than  efficient  service.”  (p.  7) 
Despite  the  current  acquisition  manage¬ 
ment  environment,  we  believe  that  con¬ 
tinued  improved  efficiencies  in  system  ac¬ 
quisition  are  possible.  Historically,  acqui¬ 
sition  efficiency  meant  procuring  the  most 
goods  and  services  for  the  least  amount 
of  money.  However,  the  recent  emphasis 
in  acquisition  is  to  find  new  and  innova¬ 
tive  ways  to  reduce  costs,  increase  effi¬ 
ciency,  and  assess  the  attainment  of  these 
goals.  Recent  public  policy  trends  such  as 
privatization,  competition,  and  process 
improvement  may  offer  insight  into  how 
efficiencies  can  be  obtained.  Next  we  will 
describe  these  policy  trends  and  discuss 
how  to  assess  their  utility  for  improving 
acquisition  management. 

Policy  Trends 


Privatization  of  Functions 

Privatization  is  the  decision  to  have  a 
private  sector  entity  perform  a  task  cur¬ 
rently  performed  by  government  employ¬ 
ees  (Donahue,  1989).  More  than  100  coun¬ 
tries  have  privatized  $445  billion  worth 
of  state-owned  assets  and  enterprises  in 


the  last  decade  (Shoop,  1995). 
Privatization  encompasses  a  range  of  ac¬ 
tivities  including  government  asset  sales, 
government  service  shutdowns,  quasi- 
govemmental  relationships,  public-pri¬ 
vate  partnerships,  and  contracting  out 
(Shoop,  1995).  The  potential  for 
privatization  is  large  considering  the  gov¬ 
ernment  currently  performs  many  func¬ 
tions  in-house  that  could  be  done  by  the 
private  sector.  Examples  include  building, 
vehicle,  and  aircraft  maintenance;  payroll; 
financial  management;  medical  care;  fire 
fighting;  and  security. 

Recent  examples  of  privatization  range 
from  employee  word  processing  training 
to  management  of  nuclear  weapons  facili¬ 
ties  and  base  operations  (Hudson,  1995). 
A  recent  report  by  DoD’s  Commission  on 
Roles  and  Missions  of  the  Armed  Forces 
(CORM)  suggests  a  greater  reliance  on  the 
private  sector  and  more  privatization  of 
defense  support  (White,  1995).  Acquisi¬ 
tion  program  managers  must  now  ask  if 
the  acquisition  function  itself  is  vulnerable 
to  privatization.  Recent  acquisition  policy 
advances  such  as  Defense  Acquisition 
Workforce  Improvement  Act  of  1 99 1  have 
made  the  field  more  specialized.  This  law 
designates  acquisition  positions,  an  acqui¬ 
sition  corps,  and  identifies  qualifications 
for  acquisition  program  managers  in  a  three- 
level  system.  These  efforts  are  designed  to 
differentiate  acquisition  personnel  as  well 
as  improve  their  performance.  Potentially, 
these  initiatives  could  help  determine  which 
functions  or  levels  of  responsibility  are  suit¬ 
able  for  privatization.  Although  a  threat  to 
acquisition  management  positions, 
privatization  can  also  present  opportunities, 
such  as  widening  the  range  of  strategy  op¬ 
tions  that  affect  military  systems  acquisition 
and  their  life-cycle  (OtfeM)  costs. 
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Competition  Between  Government  or 
Civilian  Contractors 


^^hlle  shrinking 
government  is  rely¬ 
ing  more  on  private- 
sector  contractors, 
the  process  used  to 
compete  for  goods 
and  services  remains 
bureaucratic  and 
risk  adverse/' 


In  terms  of  fiscal  policy,  allowing  com¬ 
petition  between  the  government  and  the 
private  sector  represents  an  alternative  less 
severe  than  complete  privatization  of  the 
public  entity.  Supporters  of  public-private 
competition  argue  that  it  forces  public  or¬ 
ganizations  to 
become  more 
efficient  by  re¬ 
ducing  unnec¬ 
essary  costs.  In 
addition,  it  can 
encourage  inno¬ 
vation  on  both 
sides  and  also 
push  suppliers 
to  improve  the 
quality  of  their  services.  However,  dispari¬ 
ties  in  current  accounting  methods  and 
rules  between  the  two  sectors  make  evalu¬ 
ating  costs  challenging.  These  inherent 
difficulties  in  public  and  private-sector 
competition  are  key  barriers  to  effective 
implementation.  While  shrinking  govern¬ 
ment  is  relying  more  on  private-sector 
contractors,  the  process  used  to  compete 
for  goods  and  services  remains  bureau¬ 
cratic  and  risk  adverse  (Corbin,  1995). 

The  formal  policy  facilitating  compe¬ 
tition  is  contained  in  0MB  Circular  A-76, 
“Commercial  Practices  and  Competition.” 
This  policy  establishes  guidelines  and  pro¬ 
cedures  for  the  competition  of  functions 
between  the  government  and  the  private 
sector.  Although  this  program  is  more  than 
30  years  old,  private  contractors  have  only 
recently  been  permitted  to  compete  for 
many  DoD  functions. 

The  depot-level  maintenance  of  mili¬ 


tary  equipment  is  one  area  where  compe¬ 
tition  has  gained  considerable  attention 
and  media  coverage.  A  recent  Congres¬ 
sional  Budget  Office  study  (1995)  ques¬ 
tions  whether  the  public  and  private  sec¬ 
tors  traditional  depot-level  maintenance 
roles  remain  appropriate  in  today’s  envi¬ 
ronment.  Currently,  the  private  sector  per¬ 
forms  about  32%  of  DoD’s  total  depot- 
level  maintenance.  The  total  depot  work 
being  done  by  the  private  sector  ranges 
from  3%  of  Navy  submarines  to  100%  of 
Army  aircraft  maintenance  (CBO,  1995). 
In  addition,  there  are  many  mixed  modes 
of  operation  possessing  both  public  and 
private  sector  characteristics.  One  ex¬ 
ample  is  government-owned  and  contrac¬ 
tor-operated  (COCO)  plants  or  facilities. 
Consequently,  the  extent  of  private  sector 
involvement  is  sometimes  difficult  to  de¬ 
termine. 


Process  Improvement 


Improvement  of  processes  is  another 
way  for  acquisition  program  managers  to 
increase  efficiency.  Historically,  public  fis¬ 
cal  activities  have  focused  on  expenditures 
and  zeroing  out  the  program’s  budget 
rather  than  reducing  costs  (Gansler,  1980). 
As  Donahue  (1989)  argues,  cumbersome 
rules  and  procedures  associated  with  pub¬ 
lic  procurement  contribute  to  acquisition’s 
inherent  lack  of  efficiency.  Recently,  how¬ 
ever,  the  Secretary  of  Defense  authorized 
widespread  waiver  authority  on  policies 
and  regulations  to  mitigate  many  of  these 
traditional  obstacles  to  process  improve¬ 
ment  (Perry,  1994).  With  restrictions  re¬ 
moved,  acquisition  program  managers 
have  increased  flexibility  to  implement 
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processes  improving  efficiency.  Two 
popular  process  improvement  techniques 
that  may  help  the  acquisition  program 
manager  are  business  process  reengi¬ 
neering  (BPR)  and  improved  costing 
methods. 

DoD  is  embracing  business  process 
reengineering/improvement  (BPR/I)  as  a 
key  element  in  its  effort  to  enhance  effi¬ 
ciency.  Hammer  and  Champy  (1993)  de¬ 
fine  reengineering  processes  as  the  fun¬ 
damental  rethinking  and  radical  redesign 
of  business  processes  to  bring  dramatic 
improvements  in  performance.  The 
Comptroller  General  of  the  United  States, 
Charles  A.  Bowsher,  states,  “reengi¬ 
neering  and  modem  technology  offer  huge 
opportunities  to  reduce  federal  costs  while 
also  improving  the  quality  of  government 
service”  (GAO/T-OCG-95-2,  1995,  p.  1) 
and  “we  support  these  [reengineering]  ef¬ 
forts  and  will  continue  to  evaluate  DoD’s 
progress  in  fundamentally  improving  its 
business  processes”  (GAO/T-AIMD-95- 
143,  1995,  pp.  11-12).  Acquisition  man¬ 
agement  examples  include  a  recent  Na¬ 
tional  Aeronautics  and  Space  Administra¬ 
tion  (NASA)  Jet  Propulsion  Laboratory 
(JPL)  space  mission  planning  BPR  effort 
that  took  a  string  of  25-30  functional  ac¬ 
tivities  and  clarified  them  as  three  major 
processes.  JPL  thus  refocused  themselves 
as  a  mission-driven  organization  with 
much  more  insight  into  costs  (Beyond  the 
Bottom  Line,  1 995).  At  the  MILS ATCOM 
System  Program  Office  at  Los  Angeles  Air 
Force  Base,  management  redesigned 
many  of  its  processes  around  the  adoption 
of  innovative  office  technologies  such  as 
electronic  transfer  of  data  and  video  tele¬ 
conferencing  (Kiatowski,  1994). 


Assessment  of  Policy  Alternatives 

Which  of  these  policies — privatization, 
competition  between  government  and  pri¬ 
vate  contractors,  and  process  improve¬ 
ment — may  benefit  the  acquisition  pro¬ 
gram  manager?  Several  ways  to  assess 
these  policy  options  exist. 

There  are  many  reasons  why  DoD  had 
not  earlier  considered  policy  options  us¬ 
ing  the  private  sector.  One  is  the  lack  of 
internal  transfer  prices  for  government 
goods  and  services.  Evaluating  whether  to 
build  in-house 

or  contract  out  "Evaluating  whether 
is  difficult  with-  to  build  "in-house" 
out  credible  and  or  contract  out  is 
relevant  costs  difficult  without 
from  which  to  «re€llble  and  relevant 

determine  an  in-  «•»*» 

ko  determine  an  in- 

house  cost  ba-  . 

.  ^  house  cost  basis." 


or  contract  out  "Evaluating  wheth< 
is  difficult  with-  t«»  build  "in>heuse" 
out  credible  and  or  contract  out  is 
relevant  costs  difficult  without 
from  which  to  «redible  and  relev* 

determine  an  in-  «•»*» 

k^„o^  ko  determine  an  in- 

house  cost  ba-  . 

„  -  . .  house  cost  basis." 

sis.  Further,  if 

bids  do  not  con¬ 
tain  the  same  cost  elements  or  use  the  same 
rules  to  develop  those  costs  (e.g.,  depre¬ 
ciation),  they  cannot  easily  be  compared. 
Finally,  a  relevant  unit  cost  comparison 
was  often  impossible.  In  an  effort  to  bal¬ 
ance  the  playing  field,  DoD  distributes  a 
Cost  Comparability  Handbook”  spelling 
out  adjustments  needed  to  render  its  cost 
comparable  to  private  firms  (CBO  Report, 
1995).  0MB  Circular  A-76  studies  also 
have  similar  leveling  factors  and  consid¬ 
erations. 

Private  sector  entities  rely  on  transfer 
pricing  as  a  basis  to  determine  internal 
prices  and  costs.  In  the  public  sector,  a 
revolving  fund  can  facilitate  transfer  pric¬ 
ing.  A  revolving  fund  is  a  financial  tool 
used  to  transfer  charges  between  public 
organizations.  Currently,  DoD  uses  the 
Defense  Business  Operations  Fund 
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(DBOF)  as  its  revolving  fund.  Formed  by 
the  1991  Defense  Authorization  Bill,  it 
merged  DoD’s  Stock  and  Industrial  Funds 
into  one  giant  fund  (The  Pentagon,  1993a). 
As  of  1994,  the  DBOF  contains  $85  bil¬ 
lion,  allowing  certain  organizations  to 
transfer  the  charges  for  products  between 
them  (DoD  Comptroller,  1994).  The  cost 
of  non-value-added  activities  can  also  be 
recognized  because  visible  prices  are  at¬ 
tached  to  those  activities  or  processes  and 
one  can  derive  relevant  unit  costs. 

As  currently  employed,  the  DBOF  has 
some  restrictions  that  interfere  with  the  es¬ 
tablishment  of  true  marketlike  incentives. 
One  limitation  is  the  use  of  legislatively 
established  rates,  rather  than  market  prices 
or  unit  cost,  to  determine  transfer  prices 
(0PM,  1987;  The  Pentagon,  1993b).  Leg¬ 
islative  controls  on  prices  and  lack  of 
profit  incentives  impede  the  efficient  al¬ 
location  of  resources  within  public  orga¬ 
nizations  and  prevent  true  competition.  If 
a  public  organization  were  allowed  to 
make,  retain,  spend,  and  distribute  profit, 
its  management  would  have  the  incentive 
to  improve  its  organizational  efficiency. 
With  practices  now  being  employed  by 
DBOF,  the  acquisition  manager  can  cred¬ 
ibly  assess  programmatic  decisions  using 
private  sector  practices  like  financial 
analysis  and  activity-based  costing  (ABC). 

Financial  Analysis 


The  use  of  traditional  private  sector  fi¬ 
nancial  analysis  to  compare  public  and 
private  sector  operations  is  one  means  to 
evaluate  the  use  of  different  policy  op¬ 
tions.  If  financial  analysis  suggests  a  pub¬ 
lic  entity  performs  much  less  efficiently 
than  a  comparable  private  sector  business, 


then  privatization,  competition,  or  process 
improvement  options  can  be  considered. 
In  the  past,  traditional  financial  analyses 
have  not  been  generally  performed  on 
public  functions  because  of  the  lack  of  rel¬ 
evant  data.  Recent  legislation  now  has 
made  possible  the  financial  comparisons 
of  public  and  private  sector  functions.  For 
example,  using  financial  data  directed  by 
the  CFO  Act,  several  financial  ratios  can 
be  computed.  While  the  comparisons 
might  not  be  completely  valid,  the  real¬ 
ization  that  common-size  assessments  are 
possible  is  a  significant  development  use¬ 
ful  to  acquisition  program  managers. 

Financial  ratio  analysis  is  a  private  sec¬ 
tor  technique  used  to  compare  the  opera¬ 
tions  of  firms.  This  technique  facilitates 
the  analysis  of  differently  sized  organiza¬ 
tions  within  the  same  line  of  business.  The 
use  of  ratio  measures  such  as  liquidity, 
asset  management,  and  profitability  are 
appropriate  for  this  type  of  assessment 
(Harrison  &  Horngren,  1992).  For  ex¬ 
ample,  by  examining  asset  management 
measures  such  as  turnover  (i.e.,  how  well 
an  entity  uses  its  resources  to  generate 
sales)  or  inventory  turnover  (i.e.,  how  fast 
the  entity  turns  over  its  inventory),  effi¬ 
ciency  comparisons  can  be  made.  By 
evaluating  basic  financial  ratios,  an  acqui- 
sition  program  manager  can  assess 
whether  the  decision  to  privatize,  compete, 
or  improve  a  function  or  process  has  merit, 
A  recent  example  is  the  analysis  of  retail¬ 
ing  operations  of  the  Army  and  Air  Force 
Exchange  Service  (AAFES).  By  compar¬ 
ing  the  operating  performance  of  other 
large  retailers  such  as  Wal-Mart,  one  can 
determine  what  needs  improvement.  Some 
comparisons  may  show  that  the  private 
sector  can  do  a  better  job  and  should  be 
allowed  to  do  so. 
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Although  the  emergence  of  relevant  ac¬ 
counting  data  has  potential  benefits  in 
decision  making,  privatizing  decisions 
remain  complex.  Privatization  often  may 
mean  certain  government  facilities  get  less 
use.  Local  constituents  may  object  to  the 
closing  or  reduction  of  government  facili¬ 
ties,  believing  that  their  local  economy 
will  suffer  lost  jobs  and  lower  tax  revenue. 
However,  those  losses  are  often  offset  by 
new  contractor  jobs.  The  ongoing  debate 
on  how  to  operate  depot- level  mainte¬ 
nance  is  testimony  to  this  phenomenon 
(CBO,  1995).  Perhaps  the  most  significant 
consideration  is  whether  some  functions 
should  be  exempt  from  privatization  be¬ 
cause  of  combat  capability  concerns,  na¬ 
tional  security,  and  other  potential  con¬ 
flicts  of  interest. 

Financial  analysis  can  also  help  the  ac¬ 
quisition  program  manager  improve  pro¬ 
cesses  and  become  more  efficient.  With 
the  advent  of  such  legislation  as  the  CFO 
Act  and  the  existence  of  DBOF  operations, 
for  the  first  time  data  may  be  available  that 
will  allow  program  managers  to  use  pri¬ 
vate  sector  efficiency  metrics.  Also,  finan¬ 
cial  analysis  using  concepts  such  as  oper¬ 
ating  leverage  may  illustrate  which  acqui¬ 
sition  management  approach  is  the  most 
cost  efficient  and  whether  privatization  or 
competition  of  certain  functions  is  appro¬ 
priate  (Ciccotello  &  Green,  1995). 

Activih-Based  Costing 


Privatization,  competition,  and  process 
improvement  are  facilitated  by  the  grow¬ 
ing  use  of  ABC  methods  in  the  account¬ 
ing  community.  These  methods  tie  the  to¬ 
tal  costs  of  production  or  services  more 
closely  to  the  activities  driving  the  costs 


(Johnson  &  Kaplan,  1991).  Before  DBOF 
and  ABC,  the  costs  of  many  requested 
goods  and  services  were  “hidden”  to  man¬ 
agers.  With  unit 
costs  visible  ^ 

and  relevant,  relevant  unit  costs  in 
managers  may  front  of  them^  man- 
decide  whether  tig®*'*  1*®****^ 

a  product  is  re-  whether  a  product  Is 

ally  needed.  If  really  needed  A 

costs  are  too 

high,  the  commander  or  acquisition  pro¬ 
gram  manager  can  shop  elsewhere  for  the 
product  or  do  without.  In  a  related  study, 
Eldenburg  (1994)  concludes  that  making 
costs  visible  to  physicians  actually  reduces 
health  care  costs. 

Implementing  process  improvement 
involves  the  adoption  of  advanced  cost¬ 
ing  methods.  Acquisition  program  man¬ 
agers  can  learn  by  studying  recent  devel¬ 
opments  in  the  private  sector  accounting 
community.  Knowledge  of  relevant  total 
costs  and  their  effect  on  unit  costs,  as  re¬ 
vealed  through  modem  management  in¬ 
formation  systems  and  innovative  ac¬ 
counting  practices,  may  help  managers 
become  more  efficient  by  improving  the 
tracking  accuracy  of  the  true  cost  of  pro¬ 
ducing  output  (Stevenson  &  Barnes, 
1993).  With  accurate  information  about 
unit  of  production  or  activity  costs  and  the 
identification  of  cost  drivers,  acquisition 
program  managers  can  re-evaluate  their 
acquisition  strategies.  Cost  knowledge  al¬ 
lows  them  to  discern  value-added  activi¬ 
ties  from  non-value-added  activities.  By 
eliminating  the  latter,  total  costs  can  de¬ 
crease.  A  decrease  in  total  costs  helps  an 
organization  become  more  efficient  and 
allows  the  cost  savings  to  be  used  else¬ 
where.  Having  an  idea  of  a  tme  unit  cost 
may  mean  the  survival  of  a  program.  An 
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example  might  be  the  increasing  congres¬ 
sional  concern  about  the  cost  of  one  cadet 
(unit)  at  the  Air  Force  Academy  as  com¬ 
pared  with  other  commission  sources  or 
the  extremely  large  cost  now  attached  to 
a  single  (unit)  B-2  Bomber  (McMillin, 
1990). 

Incentives 


Some  would  argue  that  without  moti¬ 
vation,  workers  have  little  incentive  to  be¬ 
come  efficient.  In  many  cases,  perverse 
incentives  may  encourage  inefficiency — 
for  example,  an  effort  to  secure  bigger 
budgets  for  non- value-added  activities.  In¬ 
centives  for  efficiency  may  come  from  the 
threat  of  having  functions  privatized  or  be 


"Incentives  for 
efficiency  may  come 
from  the  threat  of 
having  functions 
privatixed  or  be 
derived  from  re¬ 
wards  for  improve¬ 
ments  in  perfor¬ 
mance/' 


wards  for  im¬ 
provements  in 
performance. 
While  possible 
incentive  pro¬ 
grams  for  ac¬ 
quisition  pro¬ 
gram  managers 
would  be  a 


function  of  size, 
service,  and  system,  the  knowledge  that 
private  sector  acquisition  program  man¬ 
agers  may  be  easier  to  motivate  might  be 
a  key  factor  to  consider.  In  fact,  some 
argue  that  the  current  basic  approach  to 
buying  major  weapon  systems,  with  a 
government  program  office  mirroring  a 
contractor  program  office,  is  a  wasteful 
duplication  of  effort.  Perhaps  only  one 
organization  is  needed  and  it  may  in¬ 
deed  be  the  contractors.  This  approach 
to  process  improvement  would  result  in 
privatization. 


For  acquisition  program  managers, 
competition  traditionally  means  more 
choices.  It  may  also  mean  increased  effi¬ 
ciency.  If  government  and  private  contrac¬ 
tor  facilities  are  bidding  for  production 
work,  the  competition  could  lower  acqui¬ 
sition  costs.  To  the  extent  that  this  mini¬ 
mizes  the  market  power  “sole  source”  con¬ 
tractors  now  possess,  the  result  will  be  re¬ 
duced  acquisition  costs.  More  choices  for 
out-year  maintenance  work  could  also  re¬ 
duce  life-cycle  cost  estimates,  which 
would  make  a  program  more  appealing  to 
DoD  decision  makers.  Also,  top-level  ac¬ 
quisition  managers  must  relinquish  some 
direct  financial  control  to  push  decision¬ 
making  down  to  lower  levels.  For  ex¬ 
ample,  acquisition  program  managers 
could  be  given  the  power  to  make  procure¬ 
ment  decisions  and  be  held  accountable 
for  financial  performance.  Such  flexibil¬ 
ity  could  change  how  program  offices  are 
staffed  and  how  acquisition  contracts  are 
designed.  Acquisition  program  managers 
could  decide,  for  example,  to  privatize 
staff  functions  typically  done  in-house.  In 
sum,  the  result  of  enhanced  program  man¬ 
ager  power  would  be  the  use  of  more  in¬ 
novative  acquisition  strategies.  Whether 
this  change  ultimately  benefited  the  gov¬ 
ernment  would  be  an  empirical  issue  wor¬ 
thy  of  examination. 


CONCIUSION 


Three  policy  initiatives  are  sweeping 
public  sector  management:  privatization, 
competition  between  the  public  and  pri¬ 
vate  sector,  and  process  improvement.  We 
believe  that  acquisition  program  manag¬ 
ers  must  not  only  be  aware  of  these  trends, 
they  must  implement  policies  to  improve 
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efficiency  or  face  the  threat  of  privati¬ 
zation  themselves.  Private  sector  assess¬ 
ment  metrics  and  cost  development  ap¬ 
proaches  such  as  financial  analysis  and 
ABC,  are  a  means  of  comparing  the  op¬ 
erations  of  public  and  private  entities.  We 
argue  that  privatization,  competition  be¬ 
tween  government  entities  and  civilian 
contractors,  and  process  improvement  all 
represent  current  trends  in  DoD  manage¬ 


ment  community  that  should  interest  ac¬ 
quisition  program  managers.  As  current 
federal  financial  management  reform  ef¬ 
forts  continue,  the  acquisition  program 
manager  should  proactively  investigate 
and  implement  them.  If  they  do  not  em¬ 
brace  this  change,  acquisition  as  we  know 
it  surely  will  not  survive  into  the  twenty- 
first  century. 
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TUTORIAL 


ADDRESSING  RISK  MANAGEMENT  IN 
NON-DEVEIOPMENTAL  ITEMS 
ACQUISITION  PROGRAMS 


Mithael  R.  Steves 


Non-developmental  Items  (NDI)  acquisition  programs  are  enjoying  popular 
support  as  faster,  cheaper  alternatives  to  new  start  research  and  development 
programs.  Unfortunately,  DoD  policy  on  risk  management  in  NDI  programs  is 
lacking.  In  the  new,  less-restrictive  DoD  Directive  5000.1  Acquisition  Policy 
and  DoD  Regulation  5000.2-R,  the  program  manager  (PM)  is  expected  to  tailor 
risk  management  practices  to  the  needs  of  the  program.  Tailoring  DoD  risk 
management  policy  to  support  NDI  program  management  leaves  the  PM  too 
much  guesswork.  An  NDI  PM’s  risk  management  program  cannot  reasonably 
benefit  from  DoD  risk  management  guidance,  procedures,  and  tools  because 
he  or  she  Is  focused  on  new  development  program  risks  and  risk  management 
practices.  Missing  is  any  explicit  consideration  of  unique  NDI  risks  and  risk 
management  requirements.  NDI  PMs  need  more  structured  policy  and 
instruction  regarding  NDI  risk  management  for  the  streamlined,  accelerated 
NDI  acquisition  environment.  A  lesson  that  we  have  learned  is  that  we  need  a 
published  risk  management  plan  as  the  source  of  NDI  risk  management 
program  decisions  and  actions.  This  article  makes  specific  recommendations 
for  including  NDI  risk  management  in  DoD  policy. 


Today’s  era  of  defense  downsizing 
has  the  military  services  scrambling 
to  protect  research,  development, 
and  acquisition  (RD&  A)  funds  and  to  pre¬ 
serve  their  acquisition  programs.  Quality 
of  life  issues,  training  and  readiness,  and 
peacekeeping  missions  continue  to  draw 
funds  away  from  RD&A  programs  and 
modernization.  The  armed  services  and 
their  acquisition  program  managers  are 
challenged  to  justify  lengthy  and  costly 
new  research  and  development  programs 


when  commercial  or  nondevelopmental 
solutions  exist.  Risky  acquisition  strate¬ 
gies  and  mistakes  in  new  research  and  de¬ 
velopment  acquisitions  have  become 
harder  to  defend.  A  new  program’s  risks 
will  demand  close  management  to  prevent 
unanticipated  or  poorly  planned-for  events 
from  exposing  the  program  to  funding  cuts 
or  cancellation. 

To  the  rescue,  or  so  many  expect,  comes 
the  Non-developmental  Item  (NDI).  NDI 
system  acquisitions  give  PMs  and  the  gov- 
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eminent  rapid,  lower  cost  access  to  ad¬ 
vanced  technology.  NDI  acquisition  strat¬ 
egies  offer  the  promise  of  risk  mitigation 
and  a  lower  risk  means  of  meeting  the 
armed  services’  urgent  mission  needs  and 
operational  requirements.  NDI  program 
management  is  not,  however,  without  risk. 
An  NDI  program  manager  must  tailor  the 
program’s  risk  management  to  the  unique 
risks  and  uncertainties  in  NDI  system  ac¬ 
quisitions.^  NDI  risk  management  de¬ 
mands  more  than  tailoring.  Problems  arise 
with  NDI  risk  management  because  NDI 
risks  and  necessary  risk  management  pro¬ 
cesses  are  not  well  understood  or  de¬ 
scribed  in  Department  of  Defense  (DoD) 
policy,  NDI  risk  management  demands 
policy,  framework,  and  tools  to  assist  the 
PM.  An  NDI  acquisition’s  purpose  is  to 
simplify  and  accelerate  the  acquisition 
process  to  meet  the  program’s,  and  ulti¬ 
mately,  our  soldier’s  needs.  Successful 
NDI  acquisition  programs  would  benefit 
from  a  more  clearly  mapped  risk  manage¬ 
ment  policy  structure,  rather  than  broad 
expectations  of  tailoring,  to  fulfill  their 
promise  of  simple,  rapid,  and  reduced-risk 
acquisitions.  The  simple,  fast,  cheaper 
route  to  NDI  risk  management  success 
requires  good  directions  and  signposts 
advising  PMs  of  risks  and  precautionary 
measures. 


Risk  Management  and 
Mon-Peveeopmentai  Item  Overview 

DoD’s  risk  management  concepts  are 
consistent  with  those  in  industry.  DoD 
guides  on  risk  management  are  similar  to 
the  Program  Management  Institute’s 
(PMI)  body  of  knowledge  on  risk  man¬ 
agement.  The  PMI  approach  reflects  the 
general  body  of  knowledge  on  risk  and 
risk  management  that  may  be  found  in 
business  and  industry  today. 

Risk  Definition 

Webster's  3d  International  Dictionary 
defines  risk  as  “the  possibility  of  loss,  in¬ 
jury,  disadvantage,  or  destruction;  some¬ 
one  or  something  that  creates  or  suggests 
a  hazard  or  adverse  chance;  the  product 
of  the  amount  that  may  be  lost  and  that 
probability  of  losing  it  (p.  1961).  DoD 
Directive  (DoDD)  5000,1  and  DoD  Regu¬ 
lation  (DoDR)  5000.2-R  do  not  define  risk 
to  coincide  with  discussions  of  risk  man¬ 
agement  or  risk  reduction.  Risk  is  often 
characterized  by  type:  cost,  schedule,  tech¬ 
nical  performance,  supportability,  and  pro¬ 
grammatic  risk  (DSMC,  1989). 

These  five  types  of  risk  interact  to  af¬ 
fect  a  program’s  overall  performance. 


Tailoring  is  a  practice  that  means  modifying  the  acquisition  process  to  reduce  the  time  required  to  meet 
user  (soldier)  needs.  Tailoring  Is  expected  to  be  done  according  to  common  sense,  effective  business 
practice,  laws,  regulations,  and  a  program’s  anticipated  fielding  date  (DoD  Directive  5000.1, 1996).  The 
concept  of  tailoring  risk  management  is  considered  vaiuable  by  DoD  as  part  of  the  program  manager’s 
authority. 


MAJ  Micheal  R.  Steves  is  Test  and  Evaluation  Coordinator  for  the  PATRIOT  Missile  System, 
Forward  Area  Air  Defense  Systems,  and  Task  Force  XXI  Tactical  Operations  Centers  at  Head¬ 
quarters,  U.S.  Army  Test  and  Evaluation  Command,  Aberdeen  Proving  Ground,  Maryland. 
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Their  basis  is  in  the  five-phase,  four-mile- 
stone,  full-scale  development  acquisition 
cycle  found  in  the  now-rescinded  DoD 
Instruction  5000.2. 

Risk  Management 

The  DoDD  5000.1  (1996,  p.4)  defines 
risk  management  as  an  approach  that  “en¬ 
compasses  risk  identification,  mitigation, 
and  continuous  tracking,  and  control  pro¬ 
cedures  that  feed  back  through  the  pro¬ 
gram  assessment  process  to  decision  au¬ 
thorities.”  This  definition  differs  from  the 
business  definition  in  that  it  stresses  a  sys¬ 
tematic  approach  rather  than  “art  and  sci¬ 
ence.”  Risk  management  identifies  and 
evaluates  the  program  areas  vulnerable  to 
high  levels  of  uncertainty.  Its  purpose  is 
to  provide  a  means  of  comparing  risk 
management  performance  to  a  standard 
and  of  tracking  risk-related  information. 
By  doing  so,  DoDR  5000.2-R  adds  that 
risk  management  becomes  an  essential  el¬ 
ement  of  a  program’s  acquisition  strategy 
(1996,  Part  3,  p.  3). 

The  Defense  Risk  Management  Process 

A  PM’s  risk  management  process 
should  set  objectives  and  continually  as¬ 
sess  the  program  for  obstacles  that  hinder 
accomplishment  of  those  objectives.  It  al¬ 
lows  the  PM  to  formulate  alternative  courses 
of  action  and  to  make  rational  decisions  on 
monitoring  and  controlling  the  outcomes  or 
consequences  of  program  events. 


Upon  identification  of  a  program’s 
risks,  a  four-part  process  begins.  The  four 
defense  risk  management  process  compo¬ 
nents  are  planning,  assessment,  analysis, 
and  handling.  Once  these  components  are 
implemented,  the  risk  mitigation  effort  is 
underway.  Risk  mitigation  is  the  combi¬ 
nation  of  risk  handling  and  risk  controls. 
DoDD  5000.1  notes  that  technology  dem¬ 
onstrations,  prototyping,  and  test  and 
evaluation  are  the  available  techniques  to 
assess  and  handle  risk.  The  PM  should 
document  the  risk  management  program, 
process,  and  techniques.  Publication  of  a 
risk  management  plan  (RMP)  is  one 
means  of  accomplishing  this. 

Non-developmental  Items 

An  NDI  is  a  previously  developed  fed¬ 
eral,  state,  local,  or  foreign  government 
item  with  little  or  no  development  effort 
required  (5000.2-R,  Part  3,  p.5).  NDI  ac¬ 
quisition  strategies  look  to  governments 
or  commercial  vendors  as  sources  for  a 
developed  and  available  item.  DoDD 
5000.1  encourages  the  armed  services  to 
use  the  most  cost-effective  materiel  solu¬ 
tion  over  a  system’s  life  cycle.  The  first 
choice  among  materiel  alternatives  after 
buying  commercial  off-the-shelf  items  is 
to  procure  or  modify  previously  developed 
U.S.  military  or  Allied  systems  or  equip¬ 
ment  (p.  6). 

The  armed  services  in  the  past  have  de¬ 
fined  NDIs  more  specifically.^  The  Army 


^  Confusion  sometimes  results  from  the  attempt  to  distinguish  an  NDI  from  a  commercial  off-the-shelf  (COTS) 
item,  A  COTS  item  does  not  make  up  a  portion  of  NDIs.  COTS  items  are  commercial  hardware  or  software 
Items  not  yet  modified  by  the  government,  items  that  are  in  the  commercial  inventory  or  production,  that 
have  proven  their  performance  in  a  similar  environment,  that  have  an  existing  support  structure,  have  an 
internal  support  structure,  have  an  internal  configuration  that  flows  with  commercial  changes,  and  generally 
are  integrated  with  other  hardware  and  software  items  to  become  part  of  a  system  or  subsystem  capability. 
An  NDI  therefore  might  be  described  as  a  noncommercial  off-the-shelf  item. 
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version  distinguishes  NDI  in  three  catego¬ 
ries  (Quindlen,  1989): 

1 .  Category  A  (basic  NDI).  No  modi¬ 
fications  are  required. 

2.  Category  B  (NDI  adaptation).  This 
category  is  sometimes  referred  to  as 
“Ruggedized  NDI.” 

3 .  Category  C  (NDI  integration) .  This 
category  is  sometimes  referred  to  as 
"Militarized  NDI.” 

Advantages  and  Disadvantages  of  NDIs 

NDIs  are  considered  a  means  of  miti¬ 
gating  program  cost  and  schedule  risks. 
Risk  mitigation  can  be  implemented  in  a 
shortened  or  streamlined  Program  Defi¬ 
nition  and  Risk  Reduction  Phase  and  En¬ 
gineering  and  Manufacturing  Develop¬ 
ment  (EMD)  Phase  of  the  acquisition  pro¬ 
cess.  The  advantages  of  NDIs,  according  to 
the  Government  Accounting  Office  (GAO), 
include  shorter  acquisition  time,  reduced 
cost,  proven  technical  performance  or  ma¬ 
ture  technologies,  and  simplified  contract¬ 
ing  procedures  involved  in  procuring  estab¬ 
lished  products.  Yet  NDI  acquisitions  do  not 
eliminate  risk  they  may  actually  introduce 
new,  significant  risks.  These  risks  will  be 
examined  shortly. 

An  effective  NDI  acquisition  program 
must  fulfill  the  service’s  needs  by  field¬ 
ing  mature  technology  that  contributes  to 
combat  readiness  while  satisfying  the 
user’s  expectations.  The  three  features  that 
attract  substantial  DoD  and  Congressional 
support  for  NDI  or  commercial  alterna¬ 
tives  but  challenge  the  PM  are: 

I.  Meeting  user  requirements  based  on 
available  government  or  commer¬ 
cial  market  solutions. 


2.  Preparing  minimum  but  not  objec¬ 
tive  ("gold-plated”)  user  require¬ 
ments  to  support  lower  program 
costs  and  rapid  system  fielding. 

3.  Providing  DoD  with  greater  access 
to  state-of-the-art  technology  to 
keep  pace  with  changing  threats, 
emerging  technologies,  and  innova¬ 
tive  combat  systems. 

The  danger  to  NDI  programs  and  their 
effective  risk  management  is  in  forcing 
NDI  solutions  on  PMs  and  their  custom¬ 
ers  (the  user  community),  when  those 
NDIs  neither  meet  requirements  nor  tran¬ 
sition  effectively  make  the  transition  from 
current  government  or  commercial  appli¬ 
cations  to  battlefield  environments. 
Broadly  stated  DoD  risk  management 
policy  and  guidelines  then  contribute  to 
disadvantages  as  well  as  advantages  of 
buying  NDIs. 

The  disadvantages  of  NDIs  and  NDI 
acquisition  strategies  stem  from  the  new 
start  research  and  development  program 
paradigms  in  the  acquisition  community. 
A  developmental  item  PM  traditionally 
faces  cost,  schedule,  performance,  sup- 
portability,  and  programmatic  risks; 
NDI  program  risks  are  often  unique.  In 
reviewing  the  literature,  I  have  found  that 
the  following  areas  make  up  the  most 
pressing  potential  risks  facing  NDI  pro¬ 
gram  objectives:  a)  requirements,  b)  NDI 
acquisition  management  environment,  c) 
performance  specifications  and  nongov¬ 
ernment  standards  (S&S),  d)  test  and 
evaluation  (T&E),  and  e)  integrated  logis¬ 
tics  support  (ILS). 

The  NDI  PM’s  principal  disadvantage 
is  a  lack  of  both  DoD  policy  attention  paid 
to  the  above  NDI  program  risks  and  the 
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procedures  and  tools  to  manage  them.  This 
void  in  NDI-specific  risk  management 
guidance  leaves  the  PM  without  current 
references  for  structuring  a  risk  manage¬ 
ment  program  or  publishing  an  RMP. 
Meanwhile,  the  pressure  to  use  NDI  solu¬ 
tions  at  the  expense  of  a  service’s  original 
needs  and  requirements  may  yield  fielded 
items  that  aren’t  compatible  or  interoper¬ 
able  with  the  intended  operational  envi¬ 
ronment  or  battlefield  systems. 

The  Divergent  Paths 


The  crux  of  the  NDI  risk  management 
issue  is  that  the  paths  of  risk  management 
policy  and  NDI  policy  do  not  cross.  DoD 
policy  emphasizes  both  risk  management 
and  NDI  but  in  differing  directions.  Risk 
management  policy  in  the  DoD  5000  se¬ 
ries  keys  on  developmental  items  and  ac¬ 
quisition  strategies.  This  is  demonstrated 
in  the  traditional  cost,  schedule,  and  per¬ 
formance  risk  orientation  of  current  policy. 
NDI  policy  focuses  on  risk  mitigation 
gained  through  use  of  established  govern¬ 
ment  sources  of  supply.  Cost  and  sched¬ 
ule  benefits  aside,  no  description  or  in¬ 
struction  for  NDI  program  risk  manage¬ 
ment  appears  in  the  DoD  5000  series.  The 
5000.1  and  5000. 2-R  policy  voids  exist 
within  the  expectation  that  the  program 
manager  will  “tailor”  (overlap,  combine, 
or  omit)  risk  management  requirements  to 
an  NDI  acquisition  strategy. 

NDI  acquisitions  streamline  acquisition 
policy  and  particularly,  risk  management 
policy,  and  result  in  tradeoffs  between  pro¬ 
gram  objectives.  While  NDI  programs 
may  mitigate  risks  in  cost  and  schedule, 
technical  performance,  supportability,  and 
programmatic  risks  may  be  heightened  by 


the  NDI-specific  risks  listed  earlier.  The 
consequences  of  NDI-specific  risk  events 
in  these  areas  can  have  tremendous  affect 
on  cost  and  schedule  in  the  long  run. 

NDI  PMs  currently  conduct  risk  iden¬ 
tification,  planning,  analysis,  assessment, 
handling,  and  documentation  according  to 
the  5000.1  and  5000.2-R.  In  doing  so,  the 
PM  applies  developmental  acquisition  risk 
management  policy  and  procedures  focus¬ 
ing  on  traditional  developmental  program 
risks  to  nondevelopmental  systems.  The 
point  is  that  many  NDI  acquisition  PMs 
must  “think  on  the  move”  without  explicit 
directions  or  road  maps.  Without  DoD  or 
service-specific  NDI  risk  management 
policy,  instructions,  templates,  or  unoffi¬ 
cial  emphasis  in  Defense  Systems  Man¬ 
agement  College  (DSMC)  risk  manage¬ 
ment  publications  or  instruction,  PMs  will 
continue  to  rely  on  gut  feelings,  develop¬ 
mental  lessons  learned,  and  incomplete 
references.  NDI 

risk  manap  «The  crux  of  the  NDI 
ment  planning  risii  niunagemenf 

can  be  second-  Jisue  Is  lhat 
guessed  since  jiaf hs  of  risk  man- 
no  official  refer-  agement  policy  and 
ences  can  be  NDi  policy  do  not 

cited  to  justify 
risk  manage¬ 
ment  decisions.  Lack  of  published  DoD 
RMP  formats  increases  this  risk. 

There  are  reasons  for  this  policy  void. 
The  risks  in  NDI  acquisition  are  not  as 
widely  described  or  documented  as  the 
benefits.  The  absence  of  documentation 
and  description  is  due  to  lack  of  familiar¬ 
ity  by  PMs,  users,  and  DoD  overall  with 
the  NDI  acquisition  risks  in  competitive, 
commercial  (defense-  and  nondefense-re¬ 
lated)  industries.  These  industry  sources 
provide  DoD  with  a  growing  number  of 
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NDI  systems  not  initially  developed  to 
meet  a  specified  armed  service  requirement. 
The  defense  acquisition  community’s  un¬ 
familiarity  with  NDI  risks  and  risk  man¬ 
agement  means  a  lack  of  understanding 
with  commercial  risk  management  prac¬ 
tices  and  standards.  Without  formal  risk 
management  policy  and  guideline  initia¬ 
tives  by  DoD  acquisition  participants,  de¬ 
finitive  policy  and  instruction  has  insuffi¬ 
cient  support  or  momentum  to  be  officially 
implemented. 

Merging  the  Two  Paths 


The  challenge  of  DoD  is  to  bridge  the 
gap  between  developmental  acquisition 
risk  management  policy  and  practices  and 
those  that  apply  specifically  to  NDI  risks. 
Effective  application  of  NDI-based  risk 
management  policy  and  practice  can  save 
NDI  and  possibly  COTS  PMs  valuable 
time,  money,  and  manpower  resources 
while  assisting  them  in  effectively  meet¬ 
ing  program  goals  and  objectives. 

Proposed  DoD  PoLia  Revisions 

DoDD  5000.1  Risk  Management 
Policy.  DoDD  5000. 1  contains  five  areas 
which  could  be  revised  to  make  its  risk 
management  guidance  suitable  for  NDI 
PMs.  The  first  area  is  the  risk  management 
subsection  of  Part  1  (p.  4).  It  presently 
states: 

[Program  mangers]  and  other  acqui¬ 
sition  managers  shall  continually 
assess  program  risks.  Risks  must  be 
well  understood,  and  risk  manage¬ 
ment  approaches  developed,  before 
decision  authorities  can  authorize  a 
program  to  proceed  into  the  next 


phase  of  the  acquisition  process.  Risk 
management  encompasses  identifi¬ 
cation,  mitigation  and  continuous 
tracking,  and  control  procedures  that 
feed  back  through  the  program  as¬ 
sessment  process  to  decision  authori¬ 
ties. 

This  guidance  lacks  reference  to  a  de¬ 
finitive  risk  management  planning  struc¬ 
ture.  Currently,  NDI  PMs  are  expected  to 
tailor  the  DoDD  5000.1  risk  management 
guidance  to  their  NDI  systems  acquisi¬ 
tions.  The  following  could  be  added  for 
more  structural  content  to  both  NDI  and 
developmental  risk  management  planning: 

Each  milestone  decision  point  will 
include  a  review  of  the  updated  risk 
management  plan  (RMP)  and  mea¬ 
sures  taken  to  identify,  assess,  ana¬ 
lyze,  handle,  and  document  program 
risks.  Continuity  will  be  maintained 
between  the  RMP  and  discussions  of 
program  risks  in  the  Integrated  Pro¬ 
gram  Summary,  Acquisition  Plan, 
Systems  Engineering  Master  Plan, 
Test  and  Evaluation  Master  Plan,  and 
Integrated  Logistics  Support  Plan. 

The  RMP  could  be  modeled  after 
DSMC  Risk  Management  or  DSMC  Sys¬ 
tems  Engineering  Management  Guides' 
RMP  formats  and  tailored  to  the  require¬ 
ments  of  the  individual  program.  It  would 
require  the  “teaming”  of  the  user  and  the 
PM  to  integrate  a  risk  management  focus 
that  supports  operational  need  and  require¬ 
ments  development.  It  could  then  be  moni¬ 
tored  through  the  Integrated  Product  Team 
Process. 

The  second  area  is  the  Event-Oriented 
Management  subsection  of  Part  2  (p.  6), 
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which  describes  “a  rigorous,  event-ori¬ 
ented  management  process  that  empha¬ 
sizes  effective  acquisition  planning,  im¬ 
proved  and  continuous  communications 
with  users,  and  prudent  risk  management 
by  both  the  government  and  industry.” 
This  statement  overlooks  risk  areas,  es¬ 
pecially  those  in  NDI  programs:  require¬ 
ments,  NDI  acquisition  management, 
T&E,  S&S,  and  ILS. 

The  DoD  4245.7-M,  published  in  1985, 
described  the  areas  of  risk  that  jeopardize 
achievement  of  successful  cost,  schedule, 
and  performance  objectives.  It  did  not 
describe  NDI  risks  acting  as  cost,  sched¬ 
ule,  or  performance  drivers.  The  Hierar¬ 
chy  of  Materiel  Alternatives  subsection 
immediately  following  Event-Oriented 
Management  currently  doesn’t  refer  to 
them  either.  Event-oriented  managers  are 
empowered  with  the  authority  to  make 
trade  offs  between  performance  and 
schedule  for  the  sake  of  cost  objectives. 
The  authority  to  trade  off  performance  or 
schedule  to  control  program  cost  risks  can 
imply  the  acceptance  of  force  readiness 
and  doctrinal  capability  gaps.  If  no  opera¬ 
tionally  effective  replacement  systems 
exist,  then  such  trade  offs  can  have  sig¬ 
nificant  impact  on  force  training  and  com¬ 
bat  effectiveness. 

The  following  could  be  added  to  reflect 
NDI  risks: 

NDI  programs  shall  examine  and 
track  the  effects  of  requirements, 
NDI  acquisition  management.  Test 
and  Evaluation,  Specifications  and 
Standards,  and  Integrated  Logistics 
Support  risks  on  cost,  schedule,  and 
performance  objectives.  Measures  of 
performance  for  risk  handling  shall 
be  proposed  in  the  RMP  concerning 


these  risks.  Trade-offs  shall  be  con¬ 
sidered  primarily  when  current  sys¬ 
tems  exist  to  sustain  the  force  until 
the  NDI  system  is  fielded. 


The  third  area  of  interest  is  a  combined 
look  at  the  Test  and  Evaluation  and  Mod¬ 
eling  and  Simu¬ 
lation  subsec-  "The  risks  in  NDI 

tions  in  Part  2  ; 

(p  6)  The  Risk  *?• 

Assessment  and 
Management 
subsection  re¬ 
fers  to  test  and 
evaluation.  In¬ 
tegrated  testing 

and  evaluation,  including  modeling  and 
simulation,  is  used  to  identify  areas  of 
technical  risk,  to  “assess  attainment  of 


much,  anil  testing  to 
the  wrong  or  ntodi- 
fied  requirements 
using  contractor  test 
and  evaiuation 
inputs." 


technical  performance  parameters,.,  to  re¬ 
duce  the  time,  resources,  and  risks  of  the 
acquisition  process”  (p,  6).  The  Risk  As¬ 
sessment  and  Management  subsection  it¬ 
self  reads:  “To  assess  and  manage  risk, 
[program  managers]  and  other  acquisition 
managers  shall  use  a  variety  of  techniques, 
including  technology  demonstrations, 
prototyping,  and  test  and  evaluation. 

NDI  acquisition  strategies  often  include 
accelerated  and  streamlined  (depending  on 
the  quantity  and  quality  of  contractor  test 
programs  and  data)  testing  and  evaluation 
processes  or  simply  contractor  test  and 
evaluation  data  reviews.  The  risks  in  NDI 
acquisitions  are  testing  too  little,  too  much, 
and  testing  to  the  wrong  or  modified 
requirements  using  contractor  test  and 
evaluation  inputs.  The  following  could  be 
included  to  address  these  concerns: 


NDI  market  analysis  of  proposed  de¬ 
signs,  NDI  test  and  evaluation  pro- 
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grams,  and  NDI  performance  speci¬ 
fications  shall  carefully  consider 
trade  offs  in  test  and  evaluation  for 
the  sake  of  short-term  cost  and 
schedule  objectives.  Such  trade  offs 
introduce  risks  to  NDI  life-cycle 
costs  and  support.  NDI  contractors’ 
test  and  evaluation  data  shall  be 
screened  and  validated  indepen¬ 
dently  by  the  operational  test  and 
evaluation  community. 

The  fourth  area  considers  contractor  re¬ 
sponsibilities  again  from  the  Risk  Assess¬ 
ment  and  Management  subsection.  Previ¬ 
ously,  solicitation  documents  required 
contractors  to  identify  risks  and  specify 
plans  to  assess  and  eliminate  risks  or  re¬ 
duce  them  to  acceptable  levels.  Now, 
DoDD  5000. 1  directs  that  (p.  4)  ‘To  en¬ 
sure  an  equitable  and  sensible  allocation 
of  risk  between  government  and  industry, 
[program  mangers]  and  other  acquisition 
managers  shall  develop  a  contracting  ap¬ 
proach  appropriate  to  the  type  of  system 
being  acquired.” 

The  shortfall  in  this  guidance  on  con¬ 
tractor  risk  management  is  the  lack  of  par¬ 
allelism  between  the  NDI  PM’s  RMP  and 

requirements 

''The  NDI  risk  man* 

agement  efforts  contractor.  The 

should  emphasize  NDI  risk  man¬ 
teaming  and  risk-  agement  efforts 

sharing  to  relieve  should  empha- 

the  burden  of  risk  size  teaming 

managment  from  and  risk-sharing 

resting  primarily  on  to  relieve  the 

the  government."  burden  of  risk 

management 
from  resting  primarily  on  the  government. 
This  is  particularly  true  with  the  acceler¬ 
ated  nature  of  NDI  acquisitions.  Risk  re¬ 


sponsibility  in  NDI  programs  should  rest 
in  large  part  with  the  contractor,  given  the 
supposedly  mature  system  being  offered. 
The  following  could  be  added:  “Contrac¬ 
tor  RMPs  shall  be  specified  in  solicitations 
as  a  deliverable.  These  shall  be  consistent 
with  DoD  program  managers’  RMPs.  Risk 
sharing  shall  be  emphasized  in  contracts 
and  monitored  by  the  DoD-contractor 
team  through  integrated  product  teams 
(IPTs). 

The  fifth  area  combines  its  focus  on  the 
Tailoring,  IPT,  and  Management  Control 
subsections  in  Part  2  (p.  8).  The  guidance 
makes  no  mention  of  NDI  risks  even  as 
materiel  alternatives  are  repeatedly  called 
for  prior  to  beginning  a  new  major  defense 
acquisition  program.  Risk  plans,  assess¬ 
ments,  and  controls,  which  are  subject  to 
tailoring,  should  be  comprehensive  in  na¬ 
ture  and  include  the  full  spectrum  of  risks 
expected  for  a  NDI  program.  Tailoring, 
IPTs,  and  management  control  could  ad¬ 
dress  the  five  NDI-specific  risk  areas  by 
directing  that  an  RMP  be  prepared  and  re¬ 
viewed  at  each  milestone. 

The  DoDD  5000.1  risk  management 
policy  revisions  are  an  important  first  step. 
Policy  must  clearly  state  the  risk  manage¬ 
ment  requirements  and  guidelines  as  they 
specifically  affect  a  NDI  PM’s  risk  man¬ 
agement  program.  These  changes  will  pro¬ 
vide  impetus  for  revisions  to  DoDR 

5000.2- R,  which  will  be  examined  next. 

DoDR  5000.2-R  Risk  Management 
Policy  and  Procedures.  The  NDI  pro¬ 
gram  managers  need  more  definitive  and 
explicit  policy  and  procedures  in  DoDR 

5000.2- R  pertaining  to  their  risk  manage¬ 
ment  programs.  The  5000.2-R,  even  with 
its  increased  commercial  and  nondevelop- 
mental  item  emphasis,  neglects  NDI  risk 
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management.  With  increasing  competition 
for  shrinking  DoD  RD&  A  budgets  and  re¬ 
sources,  DoD  and  Congressional  risk  tol¬ 
erance  will  decline,  pressure  will  inten¬ 
sify  to  manage  programs  according  to 
sound  business  practices,  and  “safer” 
acquisitions  will  attract  more  support.  In 
these  circumstances,  explicit  NDI  risk 
management  instruction  will  become  criti¬ 
cal  to  NDI  program  management. 

Two  parts  of  DoDR  5000.2-R  could 
better  serve  NDI  PMs  with  more  explicit 
discussion  of  NDI  risks  and  risk  manage¬ 
ment  implementation  procedures.  The  first 
is  Part  3.  The  regulation  encourages  ac¬ 
quisition  program  stakeholders  to  exam¬ 
ine  the  full  range  of  mature  or  developing 
materiel  alternatives  before  initiating  a 
new  program.  Emphasis  is  placed  on  cost, 
schedule,  performance,  interoperability, 
trade  offs,  and  risk  management.  The  use 
of  IPTs  and  tailoring  are  introduced  as 
additional  major  program  considerations 
for  each  milestone  decision  point.  A 
program’s  decision  authority,  documenta¬ 
tion,  and  milestone  reviews  will  be  sub¬ 
ject  to  the  program  size,  complexity,  and 
risk,  as  well  as  the  flexibility  intended 
through  IPTs  and  tailoring. 

As  part  of  the  original  DoD  Instruction 
5000.2,  risk  assessment  formats  were  pro¬ 
vided  in  the  now-rescinded  DoD  5000.2- 
M.  The  format  description  did  more  to 
encourage  tailoring  than  to  depict  a  spe¬ 
cific  risk  assessment  format;  it  provided 
little  to  support  outlining  an  NDI  risk  as¬ 
sessment,  much  less  an  RMP.  The  current 
5000.2-R  provides  no  formats.  PMs  are 
given  wider  latitude  to  prepare  their  own 
risk  assessments  and  risk  management 
plans  and  programs.  Any  current  format 
would  be  helpful  if  it  were  to  refer  to  the 
the  old  DoD  4245. 7-M  risk  areas. 


Additionally,  with  NDI  and  streamlining 
initiatives  more  common  today,  more  de¬ 
tail  is  required  than  currently  described  or 
provided  in  those  sets  of  templates  in  the 
supportability  and  programmatic  sources 
of  risk  as  well  as  the  NDI-specific  Re¬ 


quirements,  S&S,  T&E,  ILS,  and  NDI 
program  management  environment  risk 
areas. 

In  the  5000.2-R,  milestone  0  concen¬ 
trates  on  materiel  alternatives  to  a  new 
major  defense  acquisiton  program.  The 
milestone  I,  II,  and  III  reviews,  however, 
repeatedly  focus  on  acquisition  program 
baselines  (APB),  acquisition  strategies, 
and  exit  criteria.  The  APB  content,  espe¬ 
cially  pertain- 

ing  to  cost,  is  "The  targeted  risks 

t  are  technical,  manu- 

subject  to  re- 

ftnement  during  .  ..m 

each  program  be  confrolled  before 
phase.  A  refined  |he  next  mileslene 
APB  encour-  decision  point." 

ages  risk  reduc¬ 
tion  efforts.  Risk  reduction  efforts  are 
based  on  careful  risk  assessments,  which 
are  a  basis  for  overall  cost  parameters  and 
realistic  cost  estimates.  But  no  format  is 
provided  in  the  regulation.  No  tie-in  is 
made  to  NDI  APBs  or  NDI  risk  assess¬ 
ments  and  unique  risk  reduction  require¬ 
ments.  Program  acquisition  strategies  each 
have  risk  management  as  an  essential  ele¬ 
ment.  Again,  while  NDI  programs  will 
have  their  own  acquisition  strategies  and 
risk  management  programs,  no  format  or 
tools  are  offered.  The  exit  criteria  or 
“gates”  that  the  system  must  pass  to  meet 
program  goals  and  enter  a  new  acquisi¬ 
tion  phase  are  selected  to  track  progress 
in  important  technical,  schedule,  or  man¬ 
agement  risk  areas.  No  explicit  mention 
is  made  of  NDI,  Requirements,  S&S, 
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T&E,  ILS,  and  NDI  program  management 
risk  areas,  or  their  influence  on  exit  crite¬ 
ria. 

The  second  part  of  DoDR  5000.2-R 
procedures  concerns  Part  1 .  The  milestone 
decision  point  reviews  discussed  in  DoDR 
5000.2-R  do  not  cite  one  central  document 
as  the  reference  for  a  program’s  risk  data 
or  the  planned  risk  management  program. 
Part  1  points  to  Phase  I,  Program  Defini¬ 
tion  and  Risk  Reduction,  as  the  period 
when  prototyping,  technical  demonstra¬ 
tions,  and  early  operational  assessments 
are  used  to  assess  and  reduce  risk.  The 
targeted  risks  are  technical,  manufactur¬ 
ing,  and  support  risks  that  must  be  con¬ 
trolled  before  the  next  milestone  decison 
point.  The  regulation  implies  that  these 
risk  reduction  techniques  are  done  prima¬ 
rily  in  Phase  I.  In  addition  to  emphasizing 
risk  management  as  a  program  function 
done  throughout  the  acquisition  life  cycle, 
reference  should  be  made  to  an  RMP  as 
the  source  of  documented  program  risks, 
risk  assessments  and  analysis,  risk  reduc¬ 
tion  measures,  rationale,  and  assumptions 
in  published  risk  ratings.  This  could  be 
more  effective  than  flipping  between  the 
Integrated  Program  Summary,  Acquisition 
Plan,  T&E  Master  Plan,  and  Systems  En¬ 
gineering  Master  Plan. 

The  regulation  policy  should  explicitly 
cite  the  NDI  streamlined  acquisition  cycle, 
and  the  parts  of  DoDR  5000.2-R  in  which 
detailed  descriptions  of  NDI  risks  and  risk 
management  actions  appear.  For  example, 
part  3.3.2  of  the  5000.2-R  could  feature 
the  NDI  requirements,  test  and  evaluation, 
integrated  logistics  support,  performance 
specifications  and  commercial  standards, 
and  NDI  acquisition  management  risks. 
Risks,  possible  risk  ratings  and  assessment 
considerations,  and  initial  risk  handling 


options  could  be  listed  as  program 
deliverables  at  milestone  reviews.  NDI 
risks  and  risk  management  could  then  be 
distinguished  from  the  developmental  fea¬ 
tures  in  the  DoDR  5000,2-R. 

Risk  Management  Templates.  Below 
are  areas  for  consideration  using  DoD’s 
former  DoD  4245. 7-M  risk  management 
tool,  the  risk  template.  This  would  clarify 
DoD’s  position  on  NDI  risks  and  provide 
recommended  risk  management  measures 
to  the  benefit  of  NDI  PMs,  their  programs, 
and  their  customers. 

Requirements  Risk  Template 

Areas  of  Risk: 

1.  Operational  and  design  require¬ 
ments  that  are  ill-defined  or  over¬ 
look  NDI  alternatives. 

2.  Inadequate  market  analysis  that 
contributes  to  “mix  and  match  tech¬ 
nologies”  being  required  rather  than 
verification  of  what  technology  ac¬ 
tually  exists. 

3.  Trade  offs  in  threshold  and  objec¬ 
tive  performance  requirements  that 
fail  to  meet  the  user’s  stated  need. 
Improperly  defining  the  proposed 
system’s  prospective  sources  (com¬ 
mercial,  modified  commercial,  or 
NDI)  and  required  future  design 
modifications. 

Outline  for  Reducing  Risk: 

1 .  Insert  NDI  market  analysis  into  the 
acquisition  cycle  as  part  of  Phase 
0,  Concept  Exploration  and  Devel¬ 
opment. 
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2.  Use  the  IPX  structure  to  better 
screen  and  develop  requirements. 

Timeline:  All  phases. 

Performance  Specifications  and  Non¬ 
government  Standards  Risk  Template 

Areas  of  Risk: 

1 .  Technical  performance  in  commer¬ 
cial  applications  as  specified  in 
commercial  item  descriptions 
(CIDs)  may  not  equate  to  or  explic¬ 
itly  meet  technical  performance  in 
military  applications  as  stated  in 
MIL-SPECS  and  MIL-STDS  after 
an  NDI  acquisition  strategy  is  al¬ 
ready  approved. 

2.  Performance  specifications  and 
standards  based  on  form,  fit,  and 
function  (that  allow  contractors  to 
design  solutions)  instead  of  the 
“how-to”  MIL-SPECS  and  MIL- 
STDS  used  in  design  and  manu¬ 
facturing  may  encounter  workforce 
resistance  or  complacency. 

3.  Inadequate  market  analysis  leads  to 
acceptance  of  products  having  in¬ 
sufficient  or  undocumented  techni¬ 
cal  data  or  CIDs  with  which  to  re¬ 
compete  the  procurement  for  future 
buys. 

Note:  The  DoDR  5000.2-R  states  that 
the  PM  “shall  structure  the  acquisition 
strategy  to  promote  sufficient  program  sta¬ 
bility  to  encourage  industry  to  invest,  plan, 
and  bear  risks ...  program  acquisition  strat¬ 
egies  must  analyze  industrial  capability  to 
design,  develop,  produce,  support,  and 


restart  a  program...  analysis  will  identify 
DoD  investments  needed  to  create  any 
new  industrial  capabilities  and  the  risk  of 
industry  not  providing  the  manufacturings 
capabilities  at  the  planned  cost  and  sched¬ 
ule”  (Part  3,  p.  5). 

Outline  for  Reducing  Risks: 

1 .  Specify  in  solicitations  that  CIDs  for 
meeting  user  requirements  are  a 
deliverable. 

2.  Evaluate  the  CIDs  against  MIL- 
SPECS  to  verify  their  adequacy  for 
design  and  development. 

3.  Train  and  educate  the  acquisition 
workforce  in  CIDs  and  commercial 
specifications. 

Timeline:  All  phases. 

NDI  Acquisition  Management  Risk 

Template 

Areas  of  Risk: 

1 .  Despite  streamlining,  paperwork  re¬ 
quirements,  pricing  data,  account¬ 
ing  requirements,  and  continuous 
audits  of  NDI  programs  stifle  the 
cost  and  schedule  objectives  laid  out 
in  the  acquisition  strategy. 

2.  Traditional  developmental  program 
paradigms  and  developmental  pro¬ 
gram  mindsets  continue  to  reflect  a 
cultural  resistance  toward  imple¬ 
menting  timely  and  cost-effective 
NDI  acquisition  strategies. 

3 .  Lack  of  PM-contractor  and  PM-user 
teaming  on  risk  responsibility  and 
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risk  sharing  hampers  flexibility  in 
risk  management  efforts. 

^  4.  Programmatic  micro-management 
by  stakeholders  defeats  the  benefits 
of  an  NDI  acquisition  strategy. 

Note:  DoDR  5000.2-R  discusses  cost 
management  incentives,  stating  “risk  re¬ 
duction  through  the  use  of  mature  pro¬ 
cesses  shall  be  a  significant  factor  in 
source  selection  (Part  3,  p.7)  ...  and  “the 
acquisition  strategy  shall  discuss  types  of 
contracts  ...  to  include  considerations  of 
risk  assessment,  reasonable  risk  sharing 
by  government  and  contractor(s)”  plus  the 
schedule  risk  of  using  government  fur¬ 
nished  equipment  or  government  fur¬ 
nished  information  (Part  3,  p.  8). 

Outline  for  Reducing  Risks: 

1.  Require  RMPs  as  a  contractor  de¬ 
liverable. 

2.  Require  workforce  training  and 
education  in  NDI  through  DSMC. 

Timeline:  All  phases. 

Test  and  Evaluation  Risk  Template 

Areas  of  Risk: 

1 .  Requirements  are  not  stable,  realis¬ 
tic,  or  well-understood  by  designers, 
developers,  testers,  or  managers. 

2.  Overtesting  conducted  despite  the 
presence  of  satisfactory  contractor 
test  and  evaluation  data  package. 

3.  Developmental  and  technical  test¬ 
ing  costs  are  saved  but  operational 


testing  for  operational  effectiveness 
and  suitability  may  involve  condi¬ 
tions  not  grasped  by  contractor  test¬ 
ing  program.  These  incomplete  tests 
and  data  may  be  overlooked  or 
unquestioned  in  the  accelerated 
NDI  acquisition  cycle  and  corre¬ 
sponding  accelerated  NDI  testing 
program. 

Outline  for  Reducing  Risks: 

1 .  Test  and  evaluation  data  reviews  of 
contractor  commercial  testing  pro¬ 
gram  and  results. 

2.  Demonstrations  of  the  contractor’s 
testing  process. 

3.  Modeling  and  simulation  anchored 
in  realistic,  integrated  T&E  with 
combined  DT/OT,  and  live  fire 
T&E.  Examples  are  TECOM’s 
Simulation  and  Modeling  Anchored 
in  Real  Testing  (SMART)  program 
and  Virtual  Proving  Ground  initia¬ 
tives  and  its  Combat  Synthetic  Test 
and  Training  Assessment  Range 
(STTAR)  capability  used  at  a  recent 
National  Training  Center  rotation  by 
the  1st  CAV  Division. 

Timeline:  All  phases. 

Integrated  Logistics  Support  Risk 

Template 

Areas  of  Risk: 

1.  Technical  data  packages  may  be  un¬ 
available  or  incomplete,  which  cre¬ 
ates  instability  of  spares  and  parts 
access. 
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2.  Competitive  re-procurements  of 
parts  may  not  contain  proper  incen¬ 
tives  to  attract  spares  and  parts  ven¬ 
dors, 

3.  ILS  and  a  system  life-cycle  focus 
may  be  overlooked  during  the 
requirements  development  stage. 

4.  Use  of  military  standard  and  non¬ 
standard  parts  creates  multiple  parts 
and  spares  lines. 

5.  Depot  and  repair  levels  may  not  be 
defined  in  terms  of  operational  en¬ 
vironments. 

Outline  for  Reducing  Risks: 

1 .  Define  ILS  requirements  when  de¬ 
ciding  what  category  of  NDI  the  ac¬ 
quisition  strategy  involves. 

2.  Conduct  market  analysis  of  contrac¬ 
tor  ILS  capabilities,  ILS  testing,  and 
support  demonstrations  in  the  in¬ 
tended  operational  environment  and 
conditions.^ 

3.  Specify  training  packages  and  pub¬ 
lications  as  a  contractor  deliverable. 

Timeline:  All  phases. 

These  revisions  provide  guidance  to 
NDI  PMs.  The  template  diagram,  area  of 
risk,  risk  reduction  outline,  and  a  life  cycle 


timeline  for  managing  these  risks  should 
be  included.  Clarifying  risk  management 
policy  and  upgrading  NDI  risk  manage¬ 
ment  templates  can  streamline  the  risk 
management  process  by  saving  time,  man¬ 
power,  and  the  resources  required  to  staff 
a  risk  management  program.  The  NDI 
interpretation  of  developmental  item  tem¬ 
plates  would  become  unnecessary. 

CONCIUSiONS 


The  practice  of  risk  management  does 
not  benefit  from  “cookbook”  solutions.  If 
such  solutions  existed  for  developmental 
acquisition  programs,  there  are  very  few, 
if  any,  for  NDI  acquisition  programs.  This 
is  the  NDI  PM’s  dilemma.  The  NDI  PMs 
must  adhere  to  DoDD  5000. 1  and  DoDR 
5000. 2-R  risk  management  policies  in 
their  programs,  but  the  material  is  vague 
or  must  be  tailored  to  support  NDI  pro¬ 
gram  needs.  The  situation  leaves  the  NDI 
PM  driving  a  risk  management  program 
without  the  benefit  of  signposts  or  road 
maps. 

NDI  PMs  continuously  manage  risk  as 
part  of  today’s  streamlined  and  tailored 
NDI  acquisition  environment.  They 
should  expect  and  receive  succinct,  ex¬ 
plicit  policy  and  guidelines  to  help  them 
meet  their  risk  management  and  program 
management  goals  and  objectives.  DoDD 
5000.1  could  better  serve  the  NDI  PMs 
with  risk  management  language  directed 
to  their  specific  type  of  programs  and  ac- 


^  Options  the  program  manager  can  consider  include  those  posed  in  DSMC's  NDI  acquisition  publication 

1 .  Buy  commercial  upgrades  as  they  evolve  and  become  available. 

2.  Make  a  one-time  mass  spares  purchase  to  sustain  the  duration  of  the  system’s  life  cycle. 

3.  Buy  the  technical  data  package  to  solicit  sources  of  supply  that  coincide  with  the  end  of  original  production 
and  support  by  the  original  contractor. 
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quisition  strategies.  DoDR  50(X).2-R  could 
provide  NDI  PMs  with  both  better  risk 
management  guidance  and  implementa¬ 
tion  procedures  with  NDI-based  instruc¬ 
tions,  formats  (once  found  in  DoD  5000,2- 
M),  and  tools  (once  found  in  DoD  4245.7- 
M  risk  templates  and  risk  management 
plan). 

The  NDI  program  manager’s  chal¬ 
lenges  and  program  risks  in  the  late  1990s 
are  not  adequately  reflected  in  DoD’s  mid- 


1980s  risk  management  policy,  proce¬ 
dures,  or  tools.  Risk  management  will  re¬ 
ceive  more,  not  less,  emphasis  as  an  ex¬ 
plicit  management  function.  NDI  and  NDI 
acquisition  strategies  will  continue  to 
grow  in  popular  support  as  DoD  RD&A 
budgets  are  “downsized”.  Modernization 
at  minimum  risk,  therefore,  will  require 
properly  marked  signposts  and  a  good 
road  map.  It  is  time  for  DoD  to  print  and 
distribute  those  signs  and  maps. 
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TUTORIAL 


A  HOLISTIC  MANAGEMENT  FRAMEWORK 
FOR  SOFTWARE  ACQUISITION 

Yacov  Y,  HaimeSf  Rkhard  M,  Sthooif,  and  Clyde  6.  Chittister 


In  the  face  of  the  federal  government’s  recent  downsizing  effort  and  the 
increasing  pressure  to  reduce  government  expenditures  and  improve  oversight 
of  the  use  of  public  funds,  reform  of  government  acquisition  policies  and 
practices  has  been  a  major  initiative.  While  software  acquisition  is  increasingly 
included  in  the  discussion  of  acquisition  reform,  the  concept  of  software 
acquisition  is  in  fact  a  misnomer  (even  though  it  is  included  in  the  title  of  this 
paper).  Our  government  acquires  systems,  not  exclusive  software  per  se;  these 
systems  increasingly  include  a  major  software  component. 

Software’s  critical  impact  on  system  reliability  and  performance  makes  effective 
software  acquisition  policies  and  strategies  essential.  This  article  considers 
software  acquisition  issues  being  addressed  in  an  ongoing  project  conducted 
by  the  Center  for  Risk  Management  of  Engineering  Systems  at  the  University 
of  Virginia,  and  the  Software  Engineering  institute,  Carnegie  Mellon  University. 

Because  preventive  action  is  the  key  to  successful  risk  management,  one 
must  plan  how  to  avoid  software  system  risk  early  in  the  system  life  cycle — 
before  the  software  is  purchased.  The  federal  government’s  expenditures  for 
software  products  and  services  are  tremendous,  and  government  acquisition 
procedures,  regulations,  and  results  are  readily  available  for  public  review. 
This  article  focuses  on  government  mission-critical  software  acquisition,  and 
particularly  software  acquisition  efforts  of  the  Department  of  Defense  (DoD). 
The  results  of  this  study  will  be  useful  to  other  government  agencies  as  well  as 
to  the  private  sector. 


Effective  management  of  modern, 
complex  processes  such  as  software 
acquisition  requires  capable,  mature 
direction.  Good  management  of  techno¬ 
logical  systems  must  address  the  holistic 
nature  of  the  system  in  terms  of  its  hierar¬ 
chical,  organizational,  and  functional  de¬ 
cision  making  structure;  various  time  ho¬ 


rizons;  the  multiple  decision  makers, 
stakeholders,  and  users  of  the  system;  and 
the  host  of  technical,  institutional,  legal, 
and  other  socioeconomic  conditions  that 
require  consideration.  Good  management 
also  implies  the  ability  to  identify  program 
risks,  evaluate  their  potential  adverse  im¬ 
pact,  and  effectively  incorporate  risk  con- 
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siderations  in  the  decision  making  man¬ 
agement  framework. 

The  fundamental  goal  of  this  article  is 
to  describe  the  development  of  a  quanti¬ 
tative  risk  management  framework  for 
software  acquisition,  centered  on  hierar¬ 
chical  holographic  modeling  (HHM) 
(Haimes,  1981).  This  management  frame¬ 
work  can  then  be  generalized  for  applica¬ 
tion  to  other  emerging  large-scale  systems 
and  processes. 

Here  we  build  on  current  knowledge 
and  experience  in  software  acquisition, 
software  engineering,  management,  and 
decision  making,  and  in  risk  analysis,  and 
incorporate  this  knowledge  with  the  prin¬ 
ciples  that  guide  systems  engineering.  The 
proposed  holistic  vision  of  the  software 
acquisition  process  and  the  proposed 
methodological  framework  for  the  man¬ 
agement  of  this  process  are  ultimately 
aimed  at  controlling  the  risk  of  projects’ 
cost  overruns  and  completion  schedule 
delays.  For  a  discussion  of  managing  soft¬ 
ware  technical  risk,  see  Chittister  and 
Haimes  (1994a). 


The  Approach 


Recognizing  that  software  is  only  one 
component  of  a  larger  system  (and  that 
software  is,  itself,  a  system  made  up  of 
multiple  components,  elements,  and  enti¬ 
ties),  one  must  manage  software  acquisi¬ 
tion  by  considering  the  larger  picture  and 
take  a  systemic  approach  to  resolving  the 
complex  interconnections  of  the  multiple 
participants,  activities,  events,  risks,  and 
other  process  elements.  With  the  ever-in- 
creasing  importance  and  complexity  of  the 
software  component  of  modem  systems, 
it  is  essential  that  software  acquisition  be 
addressed  in  terms  of  its  overall  system. 

We  will  not  explicitly  address  the  mul¬ 
tiple  aspects  associated  with  the  software 
acquisition  process.  Our  objective  is  in¬ 
stead  to  develop  a  modeling  framework 
that  will  enable  the  consideration  of  such 
complexities  and  interconnectedness,  and 
then  outline  the  approach  as  it  applies  to 
particular  subcomponents  of  software  ac¬ 
quisition:  the  process  vision,  composed  of 
its  multiple  stages  of  activities;  the  pro- 
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gram  consequence  vision,  which  includes 
technical  performance,  cost,  and  schedule; 
and  the  community  maturity  vision,  which 
includes  user,  customer,  contractor,  and 
technology. 

This  work  draws  on  a  holistic  represen¬ 
tation  of  such  complex  systems  and  pro¬ 
cesses  termed  hierarchical  holographic 
modeling  (HHM).  Fundamentally,  HHM 
is  grounded  on  the  premise  that  complex 
systems  and  processes,  such  as  the  soft¬ 
ware  acquisition  process,  should  be  stud¬ 
ied  using  more  than  one  single  model, 
view,  or  perspective.  HHM  possesses  a 
dual  nature:  it  is  holistic,  investigative 
paradigm,  and  a  mathematically  sound,  hi¬ 
erarchical,  multiple-objective  decision 
making  methodology.  Exploiting  the  in¬ 
herent  synergy  of  HHM's  duality  provides 
the  necessary  theoretical,  methodological, 
and  practical  foundation  for  a  risk  assess¬ 
ment  and  management  framework  for  the 
software  acquisition  (and  other  large- 
scale)  process.  The  approach  of  this  work 
(see  Figure  1 )  is  to  represent  software  ac¬ 
quisition  by  an  HHM  model,  enhance  and 
extend  HHM’s  investigative  capabilities 
for  exploring  and  modeling  the  various 
decompositions  and  submodels,  and  then 
extend  the  quantitative  capabilities  of 
HHM  for  resolving  the  conflict  and  over¬ 
lap  associated  with  the  objectives  of  the 
various  submodels. 

HHM  FOR  Software  Acquisition 

Figure  2  depicts  an  HHM  model  for 
software  acquisition.  The  six  decomposi¬ 
tions,  or  perspectives,  of  the  software  ac¬ 
quisition  HHM  indicate  the  multiple  di¬ 
mensions  associated  with  software  acqui¬ 
sition.  The  acquisition  process  requires  the 
participation  of  numerous  organizations 
and  individuals  with  specific  functions  and 


responsibilities  as  well  as  requirements  to 
coordinate  their  activities  with  the  other 
parties.  These  organizations  have  their 
own  goals  and  objectives,  which  are  of¬ 
ten  in  competition  with  each  other.  Risks 
and  uncertainties  inherent  to  the  software 
acquisition  process  complicate  the  several 
key  decisions  that,  in  turn,  affect  the  ulti¬ 
mate  software  product.  Only  by  explor¬ 
ing  the  dimensions  and  perspectives  of  the 
overall  systems  acquisition  and  properly 
coordinating  the  objectives  and  require¬ 
ments  from  each  model  perspective  can 
one  effectively  manage  the  software  ac¬ 
quisition  process. 

Software  Acquisition  Submodels. 

Software  acquisition  capability  maturity 
implies  the  existence  of,  and  adherence  to, 
a  specified,  documented,  and  repeatable 
software  acquisition  process  that  is  man¬ 
aged  through  quantitative  strategies 
(Sherer  and  Cooper  (draft)  1994).  There¬ 
fore,  prerequisite  to  a  mature  software 
acquisition  customer  community  is  the 
establishment,  analysis,  and  acceptance  of 
a  software  acquisition  process  as  well  as 
an  appropriate  quantitative  management 
framework. 

The  multiple  views  the  HHM  provides 
are  represented  by  the  various  hierarchi¬ 
cal  holographic  submodels  (HHSs),  where 
each  HHS  addresses  the  system  from  one 
particular  perspective,  or  dimension.  As 
each  perspective  may  have  its  own  unique 
representation  of  issues,  limitations,  and 
factors,  this  diversity  would  likely  lead  to 
HHSs  of  different  modeling  topology,  na¬ 
ture,  or  structure  (e.g.,  analytical  vs,  de¬ 
scriptive  models)  (see  Figure  3).  For  in¬ 
stance,  the  process  view  of  the  software 
acquisition  HHM  represents  a  progression 
of  events  or  a  sequence  of  decisions  in  the 
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HHM  MODELS 


ANALYTICAL 
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MULTIPLE-OBJECTIVE  RESOLUTION 
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Max  Tech  Performance 
Min  Development  Time 


Customer: 

Max  Tech  Performance 
Max  Customer-Contractor  Relation 
Min  Development  Time 
Min  Schedule  Deviation 


Contractor: 

Max  Profit 

Max  Future  Earnings  Potential 


Figure  1*  Quantitative  Management  Framework 


software  acquisition  process  that  may  be 
analyzed  through  process  modeling 
(Blum,  1992),  and  then  quantified  by  one 
of  many  appropriate  tools,  such  as  deci¬ 
sion  tree  methods  or  multiple-objective 
decision  tree  methods  (Haimes  et  al., 
1990a).  The  cost  element  of  the  program 
consequences  decomposition  could  be 
modeled  by  probability  distribution  analy¬ 


sis,  supported  by  analytical  software  cost 
estimation  models  (e.g.,  constructive  cost 
model  (COCOMO)  (Boehm,  1981)).  The 
software  technical  element  of  the  program 
consequence  view  may  be  quantified  in 
terms  of  one  of  several  measurable  objec¬ 
tives  (e.g.,  reliability,  availability,  main¬ 
tainability)  and  may  use  fault  tree  analy¬ 
sis  or  Markov  process  models  in  its  solu- 
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Figure  2*  Hierarchical  Holographic  Model  for  Software  Acquisition 


SOFTWARE 

ACQUiSmON 


Figure  3*  Demonstration  of  Analytic  Methods  for  HHS  Solution 
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tion  (Johnson,  1989).  Similarly,  tht  sched¬ 
ule  perspective  may  be  analyzed  through 
program  evaluation  review  technique 
(PERT)  or  related  methods  (Boehm, 
1981).  While  each  HHS  can  then  be  solved 
independently,  a  coordinated  solution  to 
the  overall  problem  must  be  resolved  at 
the  highest  level  of  the  HHM. 

In  this  article,  we  focus  on  the  process 
perspective  of  software  acquisition,  devel¬ 
oping  a  modeling  framework  that  de¬ 
scribes  the  participants,  inputs,  activities, 
decisions,  and  interrelations  of  the  vari¬ 
ous  elements  of  software  acquisition. 

Multiple  Objective  Resolution.  The 

full  value  of  structuring  the  software  ac¬ 
quisition  analysis  in  this  manner  is  real¬ 
ized  by  using  a  multiple-objective  con¬ 
struct.  Consider,  for  example,  an  over-sim¬ 
plification  of  the  objectives  of  the  end- 
user  in  a  software  acquisition  effort.  The 
user  wants  a  system  that  achieves  a  high 
level  of  technical  performance,  and  gen¬ 
erally  prefers  that  the  system  be  developed 
and  delivered  as  soon  as  possible.  These 
statements  can  be  formalized  as  F^;  maxi¬ 
mize  technical  performance,  and  F^;  mini¬ 
mize  development  time,  where  F  repre¬ 
sents  a  specific  objective.  As  Figure  1 
shows,  in  addition  to  the  user’s  multiple 
objectives,  similar  multiple  statements  can 


be  expressed  for  the  other  participants  in 
the  software  acquisition  process. 

The  multiple-objective  approach  pro¬ 
vides  a  context  for  achieving  a  “win-win- 
win”  environment  for  the  user,  customer, 
and  contractor.  When  associated  with  each 
view  of  the  hierarchical  holographic  mod¬ 
els,  this  approach  not  only  provides  a 
promising  structure  for  resolving  compe¬ 
tition  among  participants,  but  also  helps 
decison  makers  consider  system  trade  offs 
such  as  between  performance  and  cost,  or 
between  schedule  and  technology. 


Background:  Software 
AcQUisiTiON  Issues _ 

Major  issues  in  software  acquisition 
today  include  the  criticality  of  the  software 
component  in  modem  systems,  increasing 
pressure  for  reform  initiatives,  and  the 
need  for  a  less  adversarial  acquisition  en¬ 
vironment. 

Software's  Criticality 

As  computer  use  has  become  central  to 
organizational  activities  and  engineering 
system  design,  the  software  component  of 
these  systems  has  become  increasingly 
important.  That  criticality  is  well  docu- 


Figure  4.  Hierarchical  Helographic  Model  for  Software  Acquisition 
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merited  and  universally  accepted  (Boehm, 
1984;  Haimes  and  Chittister,  1993;  Blum, 
1992;  GAO,  1990).  Chittister  and  Haimes 
(1994a)  document  a  shift  in  importance 
from  hardware  to  software  within  mod¬ 
ern  systems.  Software  has  become  the 
principal  system  design  component,  as 
well  as  the  principal  factor  affecting  sys¬ 
tem  quality.  In  fact,  software  has  been 
described  as  the  “Achilles’  heel”  of  mod¬ 
em  weapon  systems  because  it  is  a  key 
determinant  of  development  schedules  and 
because  key  functions  such  as  navigation, 
enemy  detection,  and  fire  control  depend 
on  it  (GAO,  1992b).  Examples  of  system 
failures  whose  root  was  failure  of  the  soft¬ 
ware  have  been  well  publicized  (e.g., 
GAO,  1992a).  Due  to  the  continued  ex¬ 
pansion  of  software’s  commanding  role  in 
modem  systems  (and  the  budget  for  such 
systems),  the  ability  to  effectively  acquire 
and  integrate  software  into  these  systems 
will  be  increasingly  important. 

Participants  in  the  Software 
Acquisition  Process 

The  three  principal  participants,  or 
groups  of  participants,  in  an  acquisition 
endeavor  are  the  user,  the  customer,  and 
the  contractor.  In  government  acquisition, 
these  groups  rarely  constitute  single  indi¬ 
viduals,  but  each  often  comprises  one  or 
more  organizations  and  their  representa¬ 
tives.  Under  current  practice  the  user  and 
contractor  communities  generally  commu¬ 
nicate  through  the  customer  community 
(Figure  4).  Re-engineering  reform  initia¬ 
tives  of  the  acquisition  process  would  en¬ 
courage  more  direct  user-contractor  com¬ 
munication. 

In  DoD  acquisition  activities,  the  user 
community  may  be  a  major  Command 
from  one  of  the  services,  a  Unified  or  Joint 


Command,  one  of  the  military  depart¬ 
ments,  or  even  DoD  itself.  Difficulties  in 
directly  identifying  a  single  user  or  group 
of  users  are  evident.  Quite  often,  a  repre¬ 
sentative  user  is  designated  to  act  on  be¬ 
half  of  the  larger  user  organization.  In  gen¬ 
eral,  users  are  responsible  for  identifying 
and  specifying  operational  needs,  validat¬ 
ing  the  criticality  of  those  needs,  and  re¬ 
ceiving  the  completed  system. 

The  customer,  which  in  DoD  terms  re¬ 
fers  to  the  acquisition  authority,  and  more 
specifically  the  acquisition  managers  and 
program  managers,  is  the  purchasing  agent 
acting  on  behalf  of  the  user.  The  customer 
is  responsible  for  accurately  translating  the 
user’s  needs  into  the  contractual  language 
of  systems  requirements,  writing  the  con¬ 
tract  documents,  selecting  the  best  quali¬ 
fied  contractor(s),  monitoring  system  de¬ 
velopment,  accomplishing  contract  man¬ 
agement  and  negotiation  functions,  and 
conducting  system  testing  and  acceptance. 

The  contractor  must  develop  a  system 
that  meets  the  requirements,  guidelines, 
and  limits  stated  in  the  contract.  The  abil¬ 
ity  of  each  participant  to  complete  its  task 
and  effectively  coordinate  activities  with 
the  other  participants  is  central  to  a  suc¬ 
cessful  acquisition  program. 

Software  Acquisition  Research 

Software  Development.  Over  the  past 
three  decades,  software  development  prac¬ 
tices  and  processes  have  been  much  stud¬ 
ied.  Early  efforts  to  apply  and  extend  the 
practices  and  principles  of  engineering  to 
software  led  to  the  development  of  a  new 
discipline:  software  engineering.  More  re¬ 
cent  development  of  software  life  cycle 
models  (Boehm,  1981)  and  software  pro¬ 
cess  development  models  (Feiler  & 
Humphrey,  1992),  have  helped  to  bring  a 
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Table  1.  Initial  Software  Acquisition  Maturity  Model  (SAMM), 
based  on  Sherer  &  Cooper  (1994) 


LEVEL 

FOCUS 

KEY  PROCESS  AREAS 

RESULT 

5 

Optimizing 

Process 

Optimization 

Continuous  Process  Improvement 
Technology  Insertion 

Productivity  / 
&  Quality  / 

4 

Controlied 

Quantitative 

Management 

Process  Management 

Software  Quality  Management 

Defect  Prevention 

Asset  Management 

/ 

3 

Defined 

Integrated 

Project 

Management 

Software  Project  Planning  &  Mgmt 
Process  Focus 

Software  Risk  Management 

Project  Team  Coordination 

Software  Engineering  Monitoring 
Process  Assurance 

Training 

/ 

2 

Organized 

Contract 

Management 

Software  Contracting  Preparations 
Software  Contract  Initiation 
Requirements  Management 

Software  Contract  Tracking 

Software  Contract  Oversight 
Acceptance,  Transition,  &  Support 

/ 

1 

initial 

Product  and 
Resources 

/  Risk 

degree  of  standardization  and  process  im¬ 
provement  to  the  software  development 
community. 

Much  research  has  focused  on  improv¬ 
ing  the  software  development  process  (e.g., 
Humphrey  &  Kellner,  1989;  Kellner,  1991; 
Heineman  et  al.,  1994).  Business  realities 
such  as  strong  competition,  pressure  for  in¬ 
creased  profits,  and  external  regulations 
have  spurred  the  momentum  for  an  im¬ 
proved  software  development  process 
(Austin  &  Paulish,  1993).  Improving  the 
software  development  capabilities  of  soft¬ 
ware  vendors  by  improving  their  software 
development  process  maturity  is  the  fo¬ 
cus  of  the  Software  Engineering  Institute’s 


Capability  Maturity  Model  (CMM) 
(Paulk,  Weber,  Garcia,  Chrissis  &  Bush, 
1993).  This  tool  “provides  software  or¬ 
ganizations  with  guidance  on  how  to 
gain  control  of  their  process  for  devel¬ 
oping  and  maintaining  software  and 
how  to  evolve  toward  a  culture  of  soft¬ 
ware  engineering  excellence”  (Paulk  et 
al.,  1992).  Other  related  research  includ¬ 
ing  software  process  assessment,  soft¬ 
ware  metrics,  CASE  tools,  software  en¬ 
gineering,  software  quality,  concurrent 
engineering,  and  software  reliability  en¬ 
gineering  have  been  accomplished  pre¬ 
dominately  on  behalf  of  the  software 
contractor,  to  aid  in  the  actual  develop- 
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ment  of  software  and  to  provide  visibil¬ 
ity,  through  metrics,  to  the  customer. 

Software  Acquisition  Research.  Un¬ 
fortunately,  when  compared  to  software 
development  research,  relatively  little  has 
been  studied  and  written  for  the  cus¬ 
tomer’s  benefit,  i.e.,  the  development  of 
guidelines  and  instruction  of  how  to  ef¬ 
fectively  acquire  a  software  product  and 
manage  the  acquisition  effort.  Recent, 
original  work  by  Sherer  and  Cooper 
(1994)  and  parallel  research  (Baker,  Coo¬ 
per,  Corson,  &  Stevens,  1994)  have  led  to 
initial  versions  of  a  software  acquisition 
maturity  model  (S  AMM)  for  maturing  the 
acquisition  capabilities  of  the  customer 
community.  While  the  development  of  a 
SAMM  is  still  in  its  infancy,  and  revisions 
of  draft  models  are  to  be  expected,  the  ini¬ 
tial  results  appear  promising.  As  with  the 
CMM,  the  SAMM  is  both  an  evaluative 
tool  as  well  as  a  way  to  increase  a 
community’s  capability.  An  initial  version 
of  the  SAMM  (Table  1)  proposes  a  struc¬ 
ture  of  five  progressive  levels  of  maturity 
for  software  acquisition  capability,  along 
with  key  process  areas  for  each  level.  In¬ 
creasing  the  acquisition  capability  of  the 
customer  community  improves  productiv¬ 
ity  and  program  quality  while  reducing 
risk. 

The  maturity  progression  is  intended  as 
an  upward  flow:  satisfying  the  require¬ 
ments  of  one  level  leads  to  higher  level 
functions.  While  a  lower  level  organiza¬ 
tion  may  be  practicing  elements  of  a 
higher  maturity  level,  full  progression  to 
the  next  higher  level  is  contingent  on  all 
key  process  areas  being  fulfilled  (Sherer 
&  Cooper,  1994).  Maturity  in  software  ac¬ 
quisition  capability  implies  a  verified,  re¬ 
peatable,  effective  process  and  a  quanti¬ 


tative  management  framework  for  govern¬ 
ing  that  process.  At  Level  3  maturity,  the 
customer  employs  an  integrated  project 
management,  risk  management,  and  pro¬ 
cess  management  strategy  (Figure  5). 
Level  4,  quantitative  management^  re¬ 
quires  the  customer  to  set  and  monitor 
quantitative  quality  goals  for  processes 
and  products  (Sherer  &  Cooper,  1994). 
The  quantitative  management  framework 
described  here  establishes  the  necessary 
vision  and  practices  required  for  the  cus¬ 
tomer  community’s  acquisition  matura¬ 
tion. 

Software  Risk  Research.  As  in  many 
fields,  software  development  has  experi¬ 
enced  its  share  of  project  disasters.  Risk 
assessment  and  risk  management  specifi¬ 
cally  for  software  systems  is  a  relatively 
recent  area  of  research.  Boehm’s 
groundbreaking  work  (1981)  introduced 
methods  of  decision  making  under  uncer¬ 
tainty  to  this  field.  The  Software  Engineer¬ 
ing  Institute  (SEI)  of  the  Carnegie  Mellon 
University,  a  federally  funded  research  and 
development  center  with  a  broad  charter 
to  address  software  engineering  technol¬ 
ogy,  is  a  central  participant  in  current  soft¬ 
ware  risk  research.  One  of  the  SETs  fo¬ 
cus  areas  is  software  risk  management. 
The  SEI  risk  paradigm  (see  Figure  5)  de¬ 
picts  risk  assessment  and  risk  management 
as  a  process  with  several  phases — identi¬ 
fication,  analysis,  planning,  tracking,  and 
controlling  functions — which  parallel  the 
general  concepts  of  risk  identification,  risk 
quantification,  risk  analysis,  risk  manage¬ 
ment,  and  risk  mitigation. 

One  of  SETs  major  functions  has  been 
to  develop  methodologies  for  addressing 
several  of  the  software  risk  management 
phases.  The  Risk  Taxonomy-Based 
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Questionnaire  (TBQ)  is  “a  method  for  sys¬ 
tematic  and  repeatable  identification  of  risks 
associated  with  the  development  of  a  soft¬ 
ware-dependent  project”  (Carr,  Konda, 
Monarch,  Ulrich,  &  Walker,  1993).  The 
Software  Risk  Evaluation  (SRE)  method 
(Sisti  &  Joseph,  1994)  is  a  quantification 
method  for  assessing  and  analyzing  risks  for 
a  project.  A  recent  development.  Team  Risk 
Management  (TRM)  (Higueraet  al.,  1994), 
extends  risk  management  practices  to  in¬ 
clude  team-oriented  activities  involving  the 
customer  and  contractor,  where  these  groups 
apply  risk  management  methods  together. 
TRM  is  a  framework  for  cooperative  risk 
management;  it  relies  on  strengthened  risk 
communication  between  groups  and  incor¬ 
porates  the  TBQ  and  SRE  as  fundamental 
risk  analysis  and  assessment  methods. 

Acquisition  Reform 

Acquisition  reform  is  currently  a  ma¬ 
jor  initiative  within  government.  The 


administration’s  Plan  for  Economic  De- 
velopment  in  the  Technology  Sector 
(Clinton  and  Gore,  1993)  provides  a  broad 
plan  to  reinvent  the  federal  acquisition  sys¬ 
tem.  The  need  for  improvement  has  been 
expressed  across  government  sectors.  In 
a  survey  of  senior  executives  in  the  fed¬ 
eral  government,  a  majority  stated  that 
“the  procurement  process  frequently  re¬ 
sults  in  procurement  decisions  that  are 
neither  cost  effective  nor  in  the  best  inter¬ 
ests  of  the  government”  (SAMERT,  1994). 
The  Secretary  of  Defense  stated  that  “the 
existing  DoD  acquisition  system  can  be 
best  characterized  as  an  ‘industrial  era  bu¬ 
reaucracy  in  an  information  age’  ”  (Perry, 
1994). 

Although  the  defense  acquisition  sys¬ 
tem  provides  a  structured,  highly  regulated 
process  for  systems  acquisition,  the  regu¬ 
lations  and  restrictions  imposed  on  the 
process  over  time  have  often  hampered 
efforts  toward  efficiency  and  creativity.  In 
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recent  testimony  before  the  Senate  Com¬ 
mittees  on  Governmental  Affairs  and 
Armed  Services,  the  Undersecretary  of 
Defense  for  Acquisition  and  Technology 
John  Deutch,  summarized  the  problem 
(1994): 

The  system  is  too  cumbersome  and 
takes  too  long  to  satisfy  customer  re¬ 
quirements.  In  addition,  the  system 
adds  cost  to  the  product  procured. 
DoD  has  been  able  to  develop  and 
acquire  the  best  weapon  and  support 
systems,  not  because  of  the  system, 
but  in  spite  of  it.  And  they  did  so  at  a 
price — both  in  terms  of  the  sheer 
expense  to  the  nation  and  eroded 
public  confidence  in  the  DoD  acqui¬ 
sition  system. 

The  dilemma  facing  any  government 
acquisition  reform  is  how  to  provide  suf¬ 
ficient  oversight  for  the  expenditure  of 
public  funds,  with  the  least  amount  of  in¬ 
trusive  policy  and  regulation,  yet  still  ac¬ 
complish  the  acquisition  goal.  Acquisition 
reform  must  ensure  the  continued  exist¬ 
ence  of  important  safeguards  designed  to 
ensure  the  integrity  of  the  acquisition  pro¬ 
cess.  Reforms  that  reduce  regulatory  over¬ 
sight  may  possibly  increase  the  risk  of 
mismanagement  of  public  funds.  Law¬ 
makers  often  must  balance  the  risks  in¬ 
herent  in  reducing  oversight  with  the  cost 
to  industry  and  government  to  comply 
with  oversight  regulations. 

Of  particular  concern  regarding  soft¬ 
ware  acquisition  are  some  of  the  key  char¬ 
acteristics  associated  with  software  acqui¬ 
sition:  Software  evolves  rapidly,  it  is  dif¬ 
ficult  to  explicitly  define  and  specify,  ac¬ 
quisition  officials  often  lack  software  un¬ 
derstanding,  and  there  is  difficulty  in  esti¬ 


mating  project  costs  and  time  require¬ 
ments.  The  current  acquisition  process 
requires  an  average  of  16  years  to  field  a 
new  weapons  system  (Pages,  1994),  while 
software  and  computer  product  life  cycles 
are  as  short  as  1  or  2  years.  A  software 
solution  could  become  obsolete  before  it 
is  delivered. 

Other  ongoing  issues  of  software  acqui¬ 
sition  include:  incentive-based  contract¬ 
ing  vehicles  and  their  appropriate  appli¬ 
cation,  relaxation  of  “mil-spec”  require¬ 
ments  procurement,  and  a  move  to  com¬ 
mercial  off-the-shelf  (COTS)  software 
purchases.  DoD’s  acquisition  system  of 
audit  activities,  which  includes  the  possi¬ 
bility  of  criminal  sanctions  for  violating 
established  procedures  has  cultivated  a  cli¬ 
mate  of  adversarial  relationships  rather 
than  partnerships  between  customer  and 
contractor  (Defense  Science  Board,  Sep¬ 
tember  1987).  This  has  also  led  to  a  risk- 
adverse  mentality  for  the  program  man¬ 
ager  and  the  customer  community;  there 
is  no  reward  for  taking  risks  and  huge  pen¬ 
alties  for  failure.  There  is  a  strong  need 
for  a  “win-win-win”  environment  for  the 
user,  customer,  and  contractor  communi¬ 
ties  without  the  institutionalized  mistrust 
of  the  current  system. 

HoiOGRAPHic  Modeling 
FOR  Software  Acquisition 


The  dominating  attributes  of  modem, 
large-scale  systems  are  their  multidimen¬ 
sional  nature,  hierarchical  competing  ob¬ 
jectives,  multiple  participants,  a  wide  ar¬ 
ray  of  pertinent  issues  demanding  consid¬ 
eration,  and  inherent  uncertainty.  The 
complexity  of  the  software  acquisition 
process  and  the  multiplicity  of  the  parties 


65 


Acquisition  Review  Quarteriy — Winter  1997 


involved  in  that  process  from  planning,  to 
development,  to  delivery,  and  to  mainte¬ 
nance  defy  the  success  of  any  attempt  to 
represent  this  process  by  any  one  single 
model,  structure,  or  paradigm.  In  fact,  rep¬ 
resentation  within  a  single  model  of  all  the 
aspects  of  a  large-scale  system  is  so  im¬ 
practicable  as  never  to  be  seriously  at¬ 
tempted.  However,  the  inability  to  address 
this  critical  attribute  of  large-scale  systems 
is  a  major  stumbling  block.  As  Haimes 
(1981)  originally  stated, 

To  clarify  and  document  not  only  the 
multiple  components,  objectives, 
and  constraints  of  a  system  but  also 
its  welter  of  societal  aspects  (func¬ 
tional,  temporal,  geographical,  eco¬ 
nomic,  political,  legal,  environmen¬ 
tal,  sectoral,  institutional,  etc.)  is 
quite  impossible  with  a  single  model 
analysis  and  interpretation.  Given 
this  assumption  and  the  notion  that 
even  present  integrated  models  can¬ 
not  adequately  cover  a  system’s  as¬ 
pects  per  se,  the  concept  of  hierar¬ 
chical  holographic  modeling  consti¬ 
tutes  a  comprehensive  theoretical 
framework  for  systems  modeling. 

Acceptance  of  the  HHM 

Since  its  origin  in  1981,  the  HHM  has 
provided  a  general  framework  for  address¬ 
ing  the  modeling  of  complicated,  multiple 
objective  problems  of  large  scale  and 
scope.  While  not  receiving  an  abundance 
of  direct  reference  in  the  literature,  nev¬ 
ertheless  HHM’s  multivisionary  ap¬ 
proach  to  problem  definition  and  risk 
identification  has  been  widely,  although 
often  indirectly,  accepted.  For  example, 
Yeh  et  al.  (1991)  reject  the  single¬ 
model,  step-by-step  approach  in  man¬ 


aging  complex  software  engineering  de¬ 
velopment.  They  state: 

In  most  current  practice,  decisions 
are  based  on  a  one-dimensional  view 
prescribed  by  waterfall-like  models. 
This  view  consists  of  a  single  explicit 
perspective  on  a  set  of  activities  and 
their  interdependencies  and  sched¬ 
ule — which  form  an  activity  struc¬ 
ture.  In  the  waterfall  model,  the  ac¬ 
tivities  are  sequentially  scheduled 
into  phases:  requirements  analysis, 
design,  codes,  tests,  and  so  on.  Other 
models  suggest  adding  a  second  per¬ 
spective,  a  communication  structure 
...  still  models  like  the  spiral  model 
of  software  development  and  en¬ 
hancement  and  models  based  on  pro- 
cess-maturity  levels  suggest  includ¬ 
ing  process  design  and  monitoring. 

Such  an  argument,  highlighting  the 
limitations  of  a  single  model  to  capture 
the  multiple  aspects  of  a  complex  system, 
underscores  the  contributions  of  the  HHM 
approach.  Even  the  title  of  a  recent  text. 
Software  Engineering:  A  Holistic  View 
(Blum,  1992),  denotes  the  criticality  of 
considering  the  multidimensionality  of 
complex  processes  such  as  software  de¬ 
velopment.  Throughout  his  book, 
Metasystems  Methodology,  Hall  (1989) 
uses  HHM  to  recount  the  history  of  sys¬ 
tems  methodology,  and  to  distinguish  the 
varied  applied  systems  methodologies 
from  each  other.  He  states: 

History  becomes  one  model  needed 
to  give  a  rounded  view  of  our  sub¬ 
ject  within  the  philosophy  of  hierar¬ 
chical  holographic  modeling,  de¬ 
fined  as  using  a  family  of  models  at 
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several  levels  to  seek  understanding 

of  diverse  aspects  of  a  subject  and 

thus  comprehend  the  whole. 

Many  current  risk  identification  meth¬ 
ods,  evaluation  techniques,  and  issue  in¬ 
vestigation  schemes  build  on  the  general 
principles  embodied  by  the  HHM.  For 
example,  careful  examination  of  the  SEI 
taxonomy  (Carr  et  al.,  1993),  its  purpose, 
and  methodology  indicate  a  vision  that  is 
harmonious  with  HHM:  the  taxonomy  is 
hierarchical  in  structure,  is  constituted  by 
progressive  levels  of  detail  and  abstrac¬ 
tion,  provides  a  way  to  address  the  mul¬ 
tiple  dimensions  of  a  problem,  and  serves 
to  identify  areas  of  concern  in  a  software 
acquisition  endeavor.  Recognizing  the  kin- 
ship  of  these  methods  to  the  HHM 
strengthens  the  parent  methodology,  and 
further  demonstrates  the  efficacy,  appro¬ 
priateness,  and  desirability  of  the  HHM 
as  a  framework  for  analyzing  software 
acquisition  and  other  large-scale  prob¬ 
lems. 

HHM  FOR  Software  Acquisition 

The  role  of  models  is  to  represent  the 
intrinsic  and  indispensable  properties  that 
serve  to  characterize  the  system:  that  is, 
good  models  must  capture  the  essence  of 
the  system.  Clearly,  the  multidimension¬ 
ality  of  the  acquisition  process,  and  the 
large  number  of  groups,  organizations,  and 
people  of  many  disciplines  that  are  en¬ 
gaged  in  this  process  defy  the  capability 
of  any  single  model  to  represent  the  es¬ 
sence  of  the  acquisition  process.  To  over¬ 
come  the  shortfalls  of  single  planar  mod¬ 
els  and  to  identify  all  sources  of  risk  asso¬ 
ciated  with  the  software  acquisition  pro¬ 
cess,  an  HHM  framework  will  be  adopted 
here.  HHM  assumes  an  iterative  approach 


to  providing  the  structure  for  identifying 
all  risks.  If  one  fails  to  identify  a  risk 
source  with  the  current  views  of  the  HHM, 
then  expansion  of  the  model  to  include  a 
new  decomposition  is  possible.  This  pro¬ 
cess,  itself,  will  eventually  capture  all  risk 
sources. 

The  HHM  developed  here  constitutes 
the  six  subdivisions  or  perspectives  shown 
in  Figure  2.  Note  that  each  subdivision  is 
represented  within  the  HHM  framework 
by  a  separate  submodel.  This  also  implies 
that,  when  modeled,  each  of  the  six  sub¬ 
divisions  and  the  corresponding  sub-sub¬ 
divisions  will  be  represented  by  a  set  of 
objective  functions;  constraints;  and  de¬ 
cision,  state,  exogenous,  and  random  vari¬ 
ables.  Obviously  there  will  be  common 
goals  and  objectives,  as  well  as  separate 
and  possibly  conflicting  and  competing 
objectives. 

For  notational  purposes,  the  model  of  a 
subdivision  will  be  termed  hierarchical  ho¬ 
lographic  submodel  (HHS);  thus,  there  are 
six  HHSs  in  the  HHM  of  the  acquisition 
process.  Here  we  do  not  delve  into  the 
theoretical  and  methodological  grounding 
of  the  HHM  as  a  decision  making  tool  (see 
Haimes,  1981,  Haimes,  Tarvainen,  Shima, 
&  Thadathil,  1990);  rather,  we  focus  on 
the  utility  of  the  HHM  framework  as  a 
mechanism  for  the  assessment  of  risk  asso¬ 
ciated  with  the  software  acquisition  process. 

HHM  FOR  Software  Acquisition 
Risk  Identification 

HHM  has  been  used  successfully  for 
risk  identification  in  a  wide  variety  of  ap¬ 
plications  (Haimes  et  al,  1994).  Using  the 
software  acquisition  HHM  model  (Figure 
2),  a  systemic  exploration  of  the  software 
acquisition  risk  can  be  conducted  through 
the  multiple  visions  of  the  model.  As  an 
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example,  from  the  program  consequence 
perspective,  the  software  acquisition  pro¬ 
cess  may  be  divided  into  three  conse¬ 
quence  areas:  technical,  cost,  and  sched¬ 
ule. 

1.  Technical  In  a  software  context, 
software  technical  consequences  are 
concerned  with  the  quality,  preci¬ 
sion,  accuracy,  and  performance 
over  time  of  the  software. 

2.  Cost.  Refers  to  the  progranmied  and 
unexpected  expenditures  for  procur¬ 
ing  the  software  system,  along  with 
labor,  capital,  and  other  non-mon- 
etary  costs. 

3.  Schedule.  Concerns  the  establish¬ 
ment  of,  adherence  to,  and  changes 
of  a  temporal  development  plan 
from  which  systems  integration 
schedules  and  operational  deploy¬ 
ment  schedules  are  based. 


Figure  6  depicts  one  such  representa¬ 
tion  from  the  perspective  of  program  con¬ 
sequence  HHS,  focusing  on  the  cost  risks 
of  the  software  acquisition  effort — in  par¬ 
ticular,  the  cost  risks  associated  with  each 
community  (user,  customer,  contractor, 
and  technology). 

Using  the  program  consequence  per¬ 
spective  as  the  primary  vision,  one  may 
then  examine  all  such  consequences  that 
emerge  from  the  participant  communities 
(e.g..  What  effects  will  the  customer  com¬ 
munity  have  on  the  schedule?).  Robust  ap¬ 
plication  of  the  HHM  involves  a  thorough 
examination  from  multiple  combinations 
of  perspectives  the  consequences  and  fac¬ 
tors  associated  with  the  software  acquisi¬ 
tion  effort.  Such  a  comprehensive  analy¬ 
sis  produces  a  wealth  of  understanding  of 
the  strengths  and  weaknesses  associated 
with  an  acquisition  effort,  provides  a 
framework  for  devising  a  management 
plan  to  deal  with  the  identified  shortcom¬ 
ings,  and  maintains  the  holistic  perspec¬ 
tive  critical  to  program  success. 


Figure  6.  Program  Conseguence  Submodel:  Cost  Focus 
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Software  Acquisition  Process  Models 


Each  decomposition  of  the  HHM  de¬ 
picted  in  Figure  2  provides  a  unique  per¬ 
spective  for  evaluating  and  describing 
software  acquisition.  In  this  section,  we 
consider  the  process  decomposition  of  the 
software  acquisition  HHM.  Developing 
the  process  HHS  allows  one  to  focus  on 
identifying,  understanding,  and  modeling 
the  progression  of  activities  and  interre¬ 
lations  associated  with  the  software  acqui¬ 
sition  process. 

Software  Process  Modeling 

Discussion  of  the  software  process  in 
the  literature  focuses  primarily  on  soft¬ 
ware  development  processes  and  on  the 
contractor’s  role  in  those  processes.  Of¬ 
ten  referred  to  as  the  software  life  cycle, 
the  software  development  process  is  the 
collection  of  activities  that  begins  with  the 
identification  of  a  need  and  concludes  with 
the  retirement  of  the  software  product  that 
satisfies  the  need.  Traditionally,  the  soft¬ 
ware  process  has  been  described  in  terms 
of  the  “waterfall  model”  (Boehm,  1976). 
In  this  model  (which  is  an  adaptation  from 
the  hardware  development  process  model) 
there  is  a  basic  forward  flow,  or  progres¬ 
sion  of  activities  and  events.  While  the 
model  presents  a  logical,  organized  ap¬ 
proach,  its  inflexibility  in  adapting  to  the 
unique  requirements  of  modern  software 
development  has  led  many  to  believe  that 
this  model  is  discredited  (Blum,  1992). 
More  recent  representations  of  the  soft¬ 
ware  development  process  have  included 
iterative,  prototyping  activities.  For  ex¬ 
ample,  Boehm’s  spiral  model  (1988)  con¬ 
sists  of  a  series  of  learning  cycles,  with 
each  iteration  including  the  phases  of  iden¬ 


tification,  evaluation,  planning,  and  test¬ 
ing.  With  each  successive  iteration,  greater 
insight  is  gained,  and  system  development 
is  improved. 

Most  current  software  acquisition  pro¬ 
cesses  still  follow  a  waterfall  approach. 
One  major  initiative  associated  with  cur¬ 
rent  acquisition  reform  initiatives  is  modi¬ 
fying  the  existing  process  to  better  meet 
the  unique  requirements  of  software  de¬ 
velopment.  Improvements  allowing  for 
adaptive  design,  prototyping,  and  other 
iterative  development  approaches  are  be¬ 
ing  recommended  (Scientific  Advisory 
Board,  1994). 

As  software  acquisition  encompasses  a 
range  of  activities  and  concerns  far  beyond 
that  of  merely  developing  a  product,  ex¬ 
isting  process  models  are  inadequate  for 
fully  describing  the  software  acquisition 
process.  The  process  models  developed  in 
this  section  are  descriptive  in  nature:  they 
indicate  the  basic  activities,  events,  inter¬ 
relations,  and  functions  of  the  software 
acquisition  process  as  currently  practiced 
(or  as  intended  to  be  practiced).  Realiz¬ 
ing  that  analyses  of  the  process  and  pro¬ 
cess  improvement,  alone,  are  not  sufficient 
to  accomplish  the  larger  goal  of  improv¬ 
ing  software  acquisition  (Feiler,  1994), 
these  models  constitute  an  initial  vehicle 
for  examining  the  multiplicity  of  elements 
associated  with  software  acquisition. 

Software  acquisition  process  models 
provide  a  representation  of  the  progres¬ 
sion  of  interrelated  activities,  the  interac¬ 
tions  of  the  participant  communities,  the 
functions  that  each  participant  accom¬ 
plishes,  and  the  means  for  analyzing  the 
impact  of  these  actions  on  the  actual  soft¬ 
ware  acquisition  effort.  Each  stage  of  the 
model  consists  of  several  elements  or  ac¬ 
tivities  that  define  the  principal  contribu- 
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tion  of  that  phase  to  the  overall  process. 
Before  developing  a  fully  detailed  model, 
we  first  examine  the  essential  elements 
and  activities  of  the  software  acquisition 
process.  Once  these  fundamental  elements 
have  been  identified,  we  add  complexity 
and  detail  to  the  model.  Again,  the  intent 
is  to  describe  the  current  process  and  to 
work  within  that  context,  not  to  prescribe 
an  ideal  process. 

Essential  Software  Acquisition 
Process  Model 

Essence  refers  to  the  object  of  concern, 
or  inherent  nature  of  an  activity  (Brooks, 
1987).  Using  this  definition,  the  essence 
of  the  software  acquisition  process  is  find¬ 
ing  a  solution  that  meets  the  stated  need. 
By  initially  focusing  on  the  essence  of  this 
process,  we  will  have  a  context  from 
which  to  address  the  more  detailed  issues 
that  complicate  the  process’  effectiveness. 

Abstraction  of  the  software  acquisition 
process  leads  to  the  simplified  model 
shown  in  Figure  7.  This  model  takes  the 
user’s  perceived  real-world  need  as  input 
and  produces  a  solution  to  meet  that  need. 
The  model  contains  three  translation  ac¬ 


tivities,  which  together  represent  a  trans¬ 
formation  from  a  stated  need  to  a  solu¬ 
tion: 

1.  From  the  user’s  real-world  state¬ 
ment  of  need  to  a  requirements 
statement  that  details  an  intended 
solution  for  the  need. 

2.  From  the  requirements  specification 
statement  to  a  development  and 
specification  statement.  This  state¬ 
ment  includes  the  details  of  design, 
process,  and  evaluation  along  with 
the  method  of  selecting  a  contrac¬ 
tor  to  implement  the  development 
plan. 

3 .  From  the  development  statement  to 
an  actual  system  that  satisfies  the 
user’s  real-world  need. 

The  model  highlights  the  understand¬ 
ing  each  participant  community  must  have 
of  its  role  in  the  process.  As  requirements 
are  developed  from  input  originating  in 
the  user’s  domain,  the  customer  must  un¬ 
derstand  the  user’s  domain  and  then  be 


NEED 

REQUIREMENTS^ 

USER 

CUSTOMER 

DOMAIN 

DOMAIN 
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SYSTEM  ^ 

CONTRACTOR  DOMAIN 

OPERATIONAL 
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Figure  7«  Essential  Software  Acquisition  Process  Model, 
adapted  from  Blum  (1987) 
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able  to  generate  system  requirements  that 
are  usable  by  the  contractor.  These  require¬ 
ments,  however,  are  not  specific  enough 
to  provide  sufficient  detail  to  fully  define 
the  intended  system.  Therefore,  require¬ 
ments  must  be  translated  into  more  for¬ 
mal,  detailed  system  design  and  develop¬ 
ment  statements.  The  contractor’s  under¬ 
standing  of  software  tools  and  environ¬ 
ments  must  also  extend  to  an  understand¬ 
ing  of  the  implementation  domain — that 
domain  in  which  the  developed  system 
will  operate. 

The  essential  model  explains  why  the 
software  acquisition  process  is  so  compli¬ 
cated:  It  requires  the  coordinated  activi¬ 
ties  of  several  participant  communities, 
necessitates  experience  in  several  do¬ 
mains,  and  depends  on  a  series  of  diffi¬ 
cult  translation  activities.  As  software  con¬ 


tinues  to  assume  a  greater  role  in  modem 
systems,  the  difficulties  of  software  acqui¬ 
sition  become  the  overriding  difficulties 
in  systems  acquisition.  Technology  is 
changing  so  rapidly,  and  communication 
between  the  three  communities  is  getting 
increasingly  complicated,  especially  be¬ 
cause  of  DoD’s  demanding  operational 
needs  that  precipitate  the  requirement  for 
state-of-the-art  systems.  These  facts  con¬ 
stitute  the  driving  force  of  the  software 
acquisition  process, 

Detaiied  Software  Acquisition 
Process  Model 

Expanding  the  level  of  detail  included 
in  the  essential  model  leads  to  the  model 
depicted  in  Figure  8.  The  seven  stages  of 
the  detailed  model  parallel  that  depicted 
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in  Figure  7.  Stage  0  corresponds  to  the  in¬ 
put  to  the  model,  the  need  developed  by 
the  user;  stage  1  parallels  requirements 
specification;  stages  2,  3  and  4  are  an  ex¬ 
panded  treatment  of  the  development  ac¬ 
tivity;  and  stages  5  and  6  constitute  the 
system  phase  of  the  previous  model. 

In  Figure  8,  boxes  represent  an  activity 
or  function  to  be  accomplished.  Circles 
indicate  current  information,  state  of 
events,  or  other  contributing  factors  that 
cannot  be  manipulated  by  decisions  made 
in  the  acquisition  effort.  Octagons  indi¬ 
cate  output  results  that  require  monitor¬ 
ing  and  managing  and  are  affected  by  de¬ 
cisions  made  by  the  participants  in  the 
acquisition  effort.  The  three  arrows  indi¬ 
cate  the  three  translation  activities.  The 
curved  arrows  above  each  stage’s  bound¬ 
ary  indicate  the  possibility  for  iteration  in 
the  process.  While  not  institutionalized  in 
the  current  process,  reform  initiatives  and 
reengineering  activities  indicate  a  grow¬ 
ing  support  for  an  iterative  path  through 
the  acquisition  process. 

Stage  0:  Initial  Stage.  This  stage  is  a 
precursor  to  the  actual  acquisition  activi¬ 
ties;  however,  the  initial  actions  and  abili¬ 
ties  that  proceed  from  this  stage  affect  the 
balance  of  the  acquisition  effort.  Although 
this  stage  involves  both  the  user  and  cus¬ 
tomer  communities,  their  actions  at  this 
point  are  independent  of  one  another.  The 
user,  based  on  operational  experience  and 
training,  develops  an  operational  need, 
provides  a  review  process  to  formalize, 
validate,  and  prioritize  the  need,  and  for¬ 
wards  this  need  to  the  appropriate  acqui¬ 
sition  liaison  agency. 

Stage  0  also  comprises  the  baseline  lev¬ 
els  for  two  key  elements:  state-of-the-art 
software  technology,  and  customer  matu¬ 


rity.  Recent  investigation  has  shown  that 
“application-knowledgeable,  technically 
skilled  leaders  are  the  military’s  limiting 
resource  in  acquiring  today’s  computer 
technology”  (SAMERT,  1994). 

The  continued  downsizing  of  the  fed¬ 
eral  government,  and  the  DoD  in  particu¬ 
lar,  further  exacerbates  this  problem. 
Many  highly  qualified  acquisition  officials 
are  taking  advantage  of  incentives  for 
early  retirement,  realizing  that  their  skills 
are  in  great  demand  outside  of  government 
service.  Where  will  DoD’s  needed  tech¬ 
nical  experience  and  expertise  come  from? 
The  essential  key  to  an  acquisition 
program’s  success  is  the  technical  matu¬ 
rity  of  the  customer  community:  their 
knowledge  of  software  and  software  ac¬ 
quisition,  the  ability  to  understand  and 
translate  user  needs,  establish  require¬ 
ments,  develop  and  manage  contracts,  se¬ 
lect  and  monitor  appropriate  metrics,  and 
select  the  proper  contractor.  Hence  the 
need  for  continued  development  of  a  soft¬ 
ware  acquisition  maturity  model  (SAMM) 
(Sherer  &  Cooper,  1994)  aimed  at  evalu¬ 
ating  a  customer  organization’s  maturity 
level  and  providing  a  road  map  for  im¬ 
proving  its  capability. 

Stage  1.  Customer  Actions.  Follow¬ 
ing  the  user’s  development  and  validation 
of  an  operational  need,  the  customer 
agency  then  begins  the  task  of  managing 
the  acquisition  of  a  software  solution.  The 
first  translation  activity,  transforming  the 
operationally-based  language  of  the  user’s 
need  to  the  contractual  language  of  sys¬ 
tems  requirements,  is  completed  at  this 
point. 

A  requirement  is  a  “function  or  charac¬ 
teristic  of  a  system  that  is  necessary;  the 
quantifiable  and  verifiable  behaviors  that 
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a  system  must  possess  and  constraints  that 
a  system  must  work  within”  (Christel  & 
Kang,  1992).  Requirements  generally 
specify  “what”  the  system  requires  in 
terms  of  functions  and  data,  and  “how 
well”  the  system  must  perform  relative  to 
the  goals  and  objectives  of  the  system 
(Ashworth,  1989). 

The  output  of  the  requirements  identi¬ 
fication  activity  is  a  formal  statement  that 
captures  the  full  intent  of  the  user 
community’s  need  and  communicates  this 
in  appropriate  language  to  the  contractor 
community  (in  DoD,  the  result  is  called  a 
request  for  proposal  [RFP]).  In  this  light, 
requirements  analysis  is  the  bridge  be¬ 
tween  user  needs  and  system  specifica¬ 
tions  from  which  a  solution  can  be  devel¬ 
oped  (Przemieniecki,  1993).  Errors  in  re¬ 
quirements  definition  can  pass  through  un¬ 
detected  to  later  stages  of  the  acquisition 
process,  possibly  not  realized  until  a  defi¬ 
ciency  arises  at  system  implementation. 
Greater  discussion  of  the  art  and  process 
of  requirements  elicitation  and  development 
can  be  found  elsewhere  (Christel  &  Kang, 
1992;  Southwell  et  al.,  1987;  Rzepka,  1989; 
and  Fickas  &  Nagarajan,  1988). 

Stage  2.  Contractor  Actions,  While 
some  contractor  involvement  may  be  so¬ 
licited  during  requirements  generation, 
this  stage  marks  the  formal  introduction 
of  the  contractor  to  the  acquisition  pro¬ 
cess.  Candidate  contractors  conduct  the 
second  translation  activity  associated  with 
the  acquisition  effort  by  responding  to  the 
requirements  specification  with  their  de¬ 
tailed  development  plan.  This  plan  typi¬ 
cally  includes  a  description  of  the  design 
for  the  intended  system,  its  size,  structure, 
complexity,  and  other  descriptor  informa¬ 
tion.  Statements  of  technology  require¬ 


ments,  development  environment,  devel¬ 
opment  tools,  personnel,  management, 
and  other  organizational  and  technical  is¬ 
sues  are  also  included  along  with  cost  and 
schedule  figures. 

While  the  user  and  customer  may  have 
spent  considerable  time  in  accomplishing 
stages  0  and  1,  quite  often  the  contractor 
completes  stage  2  in  a  matter  of  weeks. 
Such  a  time  constraint  raises  important 
questions  concerning  the  reliability  of  the 
contractor’s  estimates.  For  instance,  how 
accurate  are  a  contractor’s  estimates  con¬ 
cerning  a  project  that  will  require  technol¬ 
ogy  beyond  current  capabilities,  relying 
on  the  development  of  yet-to-be  technol¬ 
ogy.  Other  important  questions  (Haimes 
&  Chittister,  1993)  are:  do  developers  with 
little  experience  overestimate  or  underes¬ 
timate  cost  and  schedule,  and  do  devel¬ 
opers  with  experience  overestimate  or  un¬ 
derestimate  cost  and  schedule? 

Stage  3.  Acquiring  Stage.  Some  time 
before  proposals  are  received,  the  cus¬ 
tomer  determines  the  evaluation  standards 
upon  which  all  proposals  will  be  scruti¬ 
nized  and  evaluated.  Organizational  capa¬ 
bilities,  performance  history,  cost  estimate 
practices,  as  well  as  metrics  for  evaluat¬ 
ing  other  performance  criteria  are  consid¬ 
ered.  Other  key  areas  of  interest  include 
the  contractor’s  statements  regarding  tech¬ 
nologies,  development  processes,  and  ca¬ 
pabilities. 

Generally,  during  contractor  selection, 
the  customer  is  faced  with  a  wealth  of  in¬ 
formation — some  pertinent,  some  not. 
What  is  needed  is  a  method  for  determin¬ 
ing  what  the  customer  needs  to  know,  a 
process  to  synthesize  and  filter  the  data, 
and  a  structured  process  for  using  the  in¬ 
formation  to  choose  the  contractor. 
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Stage  4.  System  Development.  In  this 
stage,  the  selected  contractor  implements 
the  development  plan  and  actually  pro¬ 
duces  the  software  system.  Most  likely, 
instead  of  a  single  contractor,  a  contrac¬ 
tor  consortium  of  teams  of  major  compa¬ 
nies  collaborate  in  the  development  of  a 
system.  The  majority  of  software  acquisi¬ 
tion  and  development  research  has  focused 
on  the  activities  of  this  stage.  The  well- 
known  waterfall  models  of  the  software 
development  life  cycle  (Royce,  1970; 
Boehm,  1981)  are  models  of  the  activi¬ 
ties  of  this  stage.  More  recent  research  on 
software  development  are  also  principally 
concerned  with  this  stage’s  events 
(Boehm,  1988;  Sage,  1992;  Feiler  & 
Humphrey,  1992;  Heineman  et  al.,  1994). 

Stage  5.  Integration.  As  system  devel¬ 
opment  progresses  to  an  initial  operating 
capability,  the  contractor,  customer,  and 
user  coordinate  the  acceptance  testing  of 
the  system.  The  success  of  this  stage 
hinges  on  the  combined  work  and  deci¬ 
sions  that  stem  from  previous  stages.  Sys¬ 
tems  integration  is  the  dominant  activity 
here;  the  components  are  integrated  with 
other  system  elements.  Increasingly,  soft¬ 
ware  is  the  vehicle  for  accomplishing  sys¬ 
tems  integration  and  assuring  system  suc¬ 
cess  (Chittister  &  Haimes,  1994b). 

At  this  stage,  previously  identified  tech¬ 
nical  and  nontechnical  risks  have  the 
greatest  likelihood  of  materializing.  These 
risks  may  have  existed  all  along,  and  come 
to  the  surface  during  the  activities  of  sys¬ 
tem  integration.  In  order  to  most  effectively 
plan  for  and  manage  the  systems  integra¬ 
tion  activity  and  the  risks  that  may  arise,  the 
customer  organization  must  make  critical 
trade  offs  between  costs,  benefits,  and  risks 
associated  with  each  policy  option. 


Stage  6.  Delivery.  Only  when  at  full 
operating  capacity  and  beyond  can  the 
value  of  risk  mitigation  and  risk  preven¬ 
tion  efforts  be  fully  realized.  Often,  selec¬ 
tion  of  a  risk  mitigation  strategy  is  based 
on  prevention  versus  correction — that  is, 
a  proactive  approach  versus  a  “wait  and 
see”  approach:  “Early  defect  fixes  are  typi¬ 
cally  two  orders  of  magnitude  cheaper 
than  late  defect  fixes,  and  the  early  re¬ 
quirements  and  design  defects  typically 
have  more  serious  operational  conse¬ 
quences”  (DoD,  1991). 

Maintenance  and  modification  initia¬ 
tives  with  their  associated  costs  and  im¬ 
pact  on  operational  capabilities  are  prin¬ 
cipal  concerns  of  this  stage.  This  stage  is 
also  the  link  for  returning  to  stage  0  in  the 
next  acquisition  effort.  If  the  customer  (or 
user,  or  contractor)  captures  the  knowl¬ 
edge  gained  through  this  acquisition  ex¬ 
perience  and  uses  it  to  increase  the  abili¬ 
ties  of  their  organization  (e.g.,  via  train¬ 
ing  or  documentation),  then  this 
community’s  maturity  level  is  increased. 
They  will  be  better  prepared  for  dealing 
with  the  next  acquisition  effort. 

Amalysis  and  Extension 
THROUGH  Process  Models 

The  process  models  developed  in  this 
section  provide  a  framework  for  a  more 
detailed  understanding  of  the  interrelations 
and  activities  associated  with  the  current 
software  acquisition  process.  Each  phase 
of  the  process  can  be  explored  in  greater 
depth  using  the  HHM  model  (Figure  2). 
The  process  models  also  provide  the  con¬ 
struct  from  which  an  iterative  software 
acquisition  process  could  be  modeled. 
These  models  are  an  effective  vehicle  for 
understanding  the  required  interconnec¬ 
tions,  mechanisms,  information,  and  ac- 
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tivities  that  must  be  included  in  such  an 
innovative  paradigm. 

IXTENDED  HHM  FOR  COORDINATED 

Submodel  Solution 


This  section  describes  two  additional 
software  acquisition  hierarchical  holo¬ 
graphic  submodels,  program  conse¬ 
quence,  and  community  maturity,  and  pre¬ 
sents  an  approach  to  resolving  the  inher¬ 
ent  conflict  in  the  analytic  modeling  as¬ 
sociated  with  these  submodels. 

Program  Consequence  HHS 

The  program  consequence  HHS  of  the 
software  acquisition  HHM  addresses  the 
need  to  synthesize  vast  amounts  of  infor¬ 
mation  concerning  the  suitability  of  a  pro¬ 
posed  system  for  meeting  an  operational 
need.  This  includes  analysis  of  the  pro¬ 
posed  system  design,  estimates  of  the 
design’s  technical  performance,  develop¬ 
ment  cost,  and  schedule  estimates,  and 


other  relevant  factors.  The  trade  offs  be¬ 
tween  performance,  cost,  schedule,  and 
the  risks  in  each  area  are  considered  in  this 
activity. 

The  multiple  factors  associated  with  the 
program  consequence  HHS  (Figure  9) 
may  be  viewed  as  submodels  of  this  HHS. 
Each  sub-submodel  may  be  quantified  and 
analyzed  by  way  of  an  appropriate  model 
(analytic  or  descriptive),  with  the  require¬ 
ment  for  an  overall  resolution  at  the  HHS 
level. 

Software  technical  performance  may  be 
quantified  in  terms  of  one  of  several  mea¬ 
surable  objectives  (e.g.,  reliability,  avail¬ 
ability,  maintainability)  and  is  best  ana¬ 
lyzed  through  fault  tree  modeling  or 
Markov  process  modeling  (Johnson,  1989; 
Kanoun  et  al.,  1993;  Tai  et  al.,  1993). 
While  several  software  cost  estimation 
models  exist  (e.g.,  Boehm,  1981 ;  Charette, 
1989;  Pressman,  1987),  each  generally 
produces  a  single-point  estimate  of  the 
projected  development  cost.  The  cost  risk- 
mitigation  approach  of  Haimes  and 


Figure  9.  Program  Consequence  HHS  Sub-submodels 
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Table  2.  Multiple  Obiectives  of  Acquisition  Process  Participants 

PARTICIPANT  OBJECTIVE 


USER 

MAXIMIZE 

MINIMIZE 

TECHNICAL  PERFORMANCE 
DEVELOPMENT  TIME 

CUSTOMER 

MAXIMIZE 

MAXIMIZE 

MINIMIZE 

MINIMIZE 

TECHNICAL  PERFORMANCE 
CUSTOMER-CONTRACTOR  RELATION 
COST 

DEVIATION  FROM  TIME  SCHEDULE 

CONTRACTOR 

MAXIMIZE 

MAXIMIZE 

PROFIT 

FUTURE  EARNINGS  POTENTIAL 

Chittister  (1993)  is  an  improved  approach, 
employing  a  probabilistic,  extreme  event 
cost  analysis  methodology  that  allows  in¬ 
corporation  of  cost  estimation  model  re¬ 
sults.  Software  project  scheduling,  most 
often  modeled  through  PERT  (Project 
Evaluation  and  Review  Technique)  or 
CPM  (Critical  Path  Method)  models 
(Charette,  1989),  also  has  probabilistic 
extensions  (Abdel-Hamid  &  Madnick 
1983;  Haimes  et  al.,  1994).  While  each  of 
the  three  sub-submodels  can  be  analyzed 
and  solved  independently,  overlapping 
objectives  and  constraints  require  that  a 
managed,  coordinated  solution  to  the  over¬ 
all  problem  be  resolved  at  the  HHS  level. 

Community  Maturity  HHS 

The  community  maturity  HHS  captures 
the  competing,  yet  overlapping  objectives 
of  the  three  participant  groups.  This  HHS 
can,  in  some  ways,  be  viewed  as  an  ex¬ 
tension  of  the  program  consequence  HHS. 
Each  participant  community  is  further  rep¬ 
resented  by  its  own  sub-HHS,  with  only 
those  consequences  applicable  to  each 


participant  group  considered  in  the 
submodels. 

In  an  acquisition  effort  the  user’s  pri¬ 
mary  objective  is  to  meet  all  of  the  opera¬ 
tional  needs.  This  objective  could  be  stated 
as  acquiring  a  system  that  maximizes  tech¬ 
nical  performance.  Generally,  this  is  not 
the  only  objective  for  the  user.  Getting  the 
system  as  soon  as  possible — minimizing 
development  time — may  also  be  impor¬ 
tant.  The  customer  has  a  similar,  but  dif¬ 
ferent  multiple-objective  problem:  to 
minimize  cost,  maximize  technical  perfor¬ 
mance,  enforce  a  contractual  time  sched¬ 
ule,  and  maximize  contractor-customer 
communication.  The  contractor  also  has 
a  multiple-objective  problem:  maximize 
profit  and  maximize  potential  for  future 
earnings  (additional  contracts).  An  ex¬ 
ample  of  these  multiple  objectives,  at  least 
one  for  each  of  the  acquisition  process  par¬ 
ticipants,  is  represented  in  Table  2. 

Obviously,  some  of  these  objectives  are 
competing  both  within  a  participant’s  indi¬ 
vidual  problem  (maximize  technical  perfor¬ 
mance  versus  minimize  cost)  and  between 
participants  (maximize  contractor’s  profit 
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versus  minimize  customer’s  cost).  A  com¬ 
mon  DoD  contracting  practice  is  to  fix 
profit  margin;  under  such  restrictions,  the 
only  way  for  a  contractor  to  increase  earn¬ 
ings  is  by  spending  more,  thereby  increas¬ 
ing  the  overall  program  cost.  A  contractor 
can  also  earn  more  with  each  deviation  and 
modification  to  the  contract.  The  more  de¬ 
viation,  the  greater  the  contractor’s  earn¬ 
ings.  These  realities  are  in  direct  compe¬ 
tition  to  the  customer’s  goals  of  minimiz¬ 
ing  cost  and  contract  deviations.  There  are 
overlaps  between  the  objectives  of  the 
three  participants,  yet  there  are  certain  ob¬ 
jectives  unique  to  each  individual  prob¬ 
lem.  Adequate  coordination  of  these 
multiobjective  problems  is  the  key  to  a 
mutually  agreeable  solution. 

As  Figure  10  shows,  analytic  method¬ 
ologies  appropriate  for  analyzing  each  of 
the  submodel  objectives  may  not  neces¬ 
sarily  assume  the  same  form.  At  least  one 


objective,  the  customer’s  desire  for  good 
relations  with  the  contractor,  may  be 
unrepresentable  mathematically.  Some 
indication  concerning  this  objective,  how¬ 
ever,  may  be  analyzed  through  a  classifi¬ 
cation  model  based  on  HHM  analysis  of 
the  characteristics  and  capabilities  of  the 
particular  contractor.  Each  sub-HHS 
model  is  itself  a  multiple  objective  model, 
and  the  result  of  the  three  participant  com¬ 
munity  solutions  must  be  resolved  at  the 
overall  HHS  level. 

Similar  multiobjective  problems  can  be 
derived  for  each  HHS  of  the  HHM.  For 
instance,  maximizing  technical  perfor¬ 
mance  may  include  the  subobjectives  of 
maximizing  reliability,  number  of  compu¬ 
tations  per  second,  system  availability,  or 
some  other  system  performance  feature. 
To  a  certain  extent,  each  of  these 
subobjectives  is  in  conflict  with  the  oth¬ 
ers — for  example,  a  system  designed  for 


Figure  10*  Community  Maturity  HHS 
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computational  speed  may  not  be  ex¬ 
tremely  reliable.  Including  cost  minimi¬ 
zation  as  a  subobjective  may  introduce 
additional  conflicts  (e.g.,  a  highly  reliable 
system  is  generally  not  the  low-cost  op¬ 
tion).  The  practical  reality  of  software  ac¬ 
quisition  demonstrates  that  it  must  be  de¬ 
scribed  in  terms  of  multiple,  multiob¬ 
jective  problems  that  are  conflicting,  pos¬ 
sible  overlapping,  and  exhibit  a  hierarchi¬ 
cal  structure. 

HHM:  Model  Management  for 
Submodel  Conflict  Resolution 

The  quantitative  framework  described 
in  this  paper  is  founded  on  the  synergistic 
coupling  of  the  HHM  and  process  mod¬ 
els,  and  the  incorporation  of  other  appro¬ 
priate  models,  methods,  and  tools  to  ef¬ 
fectively  analyze  and  support  decision 
making  throughout  the  acquisition  pro¬ 
cess.  Figure  1  shows  a  representation  of 
this  framework.  While  each  of  the  many 
methods  and  tools  has  been  designed  for 
its  own  unique  purpose,  each  may  give  a 
greater  contribution  when  coordinated 
with  the  results  of  other  methods.  A  holis¬ 
tic  vision  ensures  that  methodologies  are 
not  employed  for  their  own  sake  (sublevel 
optimization),  but  that  each  contributes  to 
the  overall  system  goals  and  objectives. 
In  this  manner,  we  achieve  analytic  pro¬ 
gression  through  the  complex  acquisition 
process. 

The  field  of  model  management,  a 
growing  area  of  study,  recognizes  that 
complex  problems  rarely  can  be  solved  by 
a  single  model  encompassing  all  problem 
aspects.  Solutions  to  such  problems  often 
require  the  integration  of  multiple  mod¬ 
els  each  addressing  a  specific  aspect  of  the 
problem  (Mitra  &  Dutta,  1994).  Model 
management  methodologies  provide  ap¬ 


proaches  for  combining  several  models 
into  an  integrated  model  that  is  sufficient  to 
solve  a  given  problem  (Basu  &  Blanning, 
1994).  While  several  approaches  have  been 
proposed  (e.g,,  database  management,  arti¬ 
ficial  intelligence,  conceptual  graphs),  the 
most  promising  and  easily  implemented 
are  those  of  the  graphical  approach.  These 
methods  provide  a  framework  for  model 
composition  by  identifying  models  that 
may  be  combined  into  a  composite  model 
(Muhanna  &  Pick,  1994).  The  HHM  can 
appropriately  be  considered  as  a  model 
management  methodology;  resolution  of 
the  higher  level  model  requires  the  coor¬ 
dinated  solution  of  multiple  submodels. 
HHM’s  unique  handling  of  overlapping 
objectives  and  program  constraints  is  par¬ 
ticularly  desirable.  While  the  original 
HHM  assumes  that  each  submodel  has  a 
mathematical  programming  formulation, 
relaxing  this  requirement  provides  a 
means  for  resolving  the  more  realistic 
problem  where  submodels  have  diverse 
analytic  constructs. 

HHM  provides  the  framework  for  rep¬ 
resenting  the  software  acquisition 
submodels  in  terms  of  hierarchical 
multiobjective  decision  models  (MODM). 
The  MODM  approach  provides  a  context 
for  a  “win-win-win”  environment,  as  a 
solution  that  is  mutually  acceptable  to  all 
three  participants  would  be  found  in  the 
set  of  nondominated  solutions  to  the  co¬ 
ordinated  multiobjective  problem 
(Chankong  &  Haimes,  1983).  The 
multiobjective  modeling  approach  also 
provides  a  structure  for  resolving  compe¬ 
tition  between  issues,  such  as  the  trade  off 
between  performance  and  cost,  or  between 
schedule  versus  technology. 

Formulation  of  a  hierarchical  multiob¬ 
jective  problem,  taken  from  the  frame- 
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work  of  the  HHM,  implies  the  formal 
evaluation  of  model  elements  and  data 
requirements  (random  variables,  decision 
variables,  state  variables,  functional  rela¬ 
tionships,  etc.).  Such  a  comprehensive 
effort  not  only  establishes  an  overall  ana¬ 
lytical  framework  for  reaching  an  agree¬ 
able  solution,  but  also  provides  greater 
insight  and  understanding  as  to  the  inter¬ 
relationships  and  structure  of  the  software 
acquisition  process. 

When  a  purely  analytic  approach  for 
resolving  HHS  conflicts  is  not  possible,  it 
may  be  necessary  to  consider  conflict  reso¬ 
lution  strategies  (Fraser  &  Hipel,  1984; 
Raiffa,  1982),  trade  off  methodologies 
(Chankong  &  Haimes,  1983),  and  nego¬ 
tiation  strategies  (Nierenberg,  1978).  The 
community  maturity  and  program  conse¬ 
quence  problems  described  above  are  not 
ones  of  simple  multiple-objective  resolu¬ 
tion  due  to  the  overlapping  and  coordina¬ 
tion  inherent  in  the  HHM/HHS  struc¬ 
ture — therefore  application  of  conflict 
resolution  strategies  must  consider  the 
unique  characteristics  of  hierarchical, 
overlapping,  and  yet  conflicting  problems. 
Other  possible  approaches  to  be  explored 
include  some  variation  of  a  weighting 
scheme  (Chankong  &  Haimes,  1983), 
where  each  HHS  or  even  each  objective 
is  assigned  a  weight — directly  or  by  a  pair¬ 
wise  comparison — to  determine  prefer¬ 
ence  for  each  objective  and  submodel.  The 
analytic  hierarchy  process  (AHP)  (Saaty, 
1990)  may  be  useful  for  such  an  approach. 
Other  trade  off  methods,  such  as  the  sur¬ 
rogate  worth  trade  off  (SWT)  method 
(Chankong  &  Haimes,  1983)  may  also 
prove  useful. 


Conclusion 


The  holistic  approach  to  software  ac¬ 
quisition  management  presented  in  this 
paper  provides  a  theoretical,  as  well  as 
methodological  approach,  for  a  maturing 
software  acquisition  community.  This  ap¬ 
proach  compliments  the  maturity  progres¬ 
sion  described  in  the  SAMM.  Figure  11 
summarizes  the  relationship  between  man¬ 
agement  maturity  and  quantitative  risk 
management  methods.  With  the  SAMM, 
progression  to  higher  maturity  levels  as¬ 
sumes  the  continuation  of  all  lower  level 
activities  and  methods,  while  adding  new 
functions  and  processes.  Similarly,  with 
the  quantitative  risk  management  frame¬ 
work,  holistic  models  build  on  the  descrip¬ 
tive  foundation  of  the  process  models; 
used  together  they  provide  even  greater 
information  and  insight.  As  with  the 
SAMM,  lower  level  organizations  may 
make  use  of  some  of  the  higher  level  meth¬ 
ods,  but  have  not  employed  all  the  func¬ 
tions  required  to  fully  progress  to  that 
higher  level.  The  focus  of  each  analytic 
method  parallels  the  focus  of  the  related 
management  level. 

Here  we’ve  detailed  a  holistic  approach 
to  the  analysis  of  the  software  acquisi¬ 
tion  process  and  to  the  development  of 
a  quantitative  management  framework 
needed  for  the  maturing  of  the  software 
acquisition  customer  community.  This 
framework  is  founded  on  the  multiple 
visions  of  the  HHM  and  the  temporal 
progression  of  activities  captured  in  the 
process  models.  It  provides  the  means 
for  incorporating  appropriate  analytic 
models  and  methodologies  for  a  sys¬ 
temic  approach  to  quantitative  manage¬ 
ment  of  software  acquisition.  The  com¬ 
peting  interests  of  the  participants,  con- 
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Figure  11.  SAIVIM  and  Quantitative  Management  Framework  Comparisons 


flicting  performance  measures,  and  un-  tiple-objective  approach  for  analyzing  and 
certain  system  requirements  make  a  mul-  resolving  these  conflicts  appropriate. 
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REENGINEERING  THE  RFP  PROCESS 
THROUGH  KNOWIEDGE-BASED  SYSTEMS 

Dr.  Mark  B,  Nissen 


Acquisition  plays  a  vital  role  in  national  defense  of  the  United  States.  But, 
defense  acquisition  can  be  quite  difficult,  and  few  would  argue  that  current 
acquisition  processes  perform  in  an  optimal  manner.  The  magnitude  of  change 
mandated  by  FASA  and  DAWIA,  along  with  changes  mandated  from  within 
DoD,  is  such  that  “business  as  usual”  no  longer  represents  a  viable  alternative 
for  acquisition  management.  The  military  procurement  process  in  general 
requires  reengineering,  and  the  research  described  in  this  paper  focuses  upon 
the  Request  for  Proposal  (RFP)  subprocess  in  particular.  The  use  and  utility  of 
a  knowledge-based  system  to  support  process  redesign  are  demonstrated, 
and  insight  is  provided  into  the  potential  of  Al-based  technologies  to 
dramatically  improve  military  procurement.  The  results  provide  the  basis  for  a 
number  of  conclusions  that  are  important  for  the  acquisition  professional,  and 
establish  an  agenda  for  future  research. 


Acquisition  plays  a  vital  role  in  na¬ 
tional  defense  of  the  United  States. 
The  armed  forces  depend  on  the 
acquisition  of  high-quality  and  reliable 
weapon  systems,  equipment,  support,  and 
services  for  their  defense  missions,  in 
peacetime  as  well  as  during  periods  of  in¬ 
ternational  tension  and  conflict.  But  get¬ 
ting  what  is  needed  can  be  quite  difficult. 
Because  many  of  these  systems  are  ex¬ 
ceedingly  sophisticated,  complex,  and  ex¬ 
pensive,  their  procurement  is  complicated 
and  involved.  Further,  those  whose  job  it 
is  to  acquire  systems  and  services  work  in 
an  environment  of  arcane  laws,  regula¬ 
tions,  policies,  and  procedures,  for  which 
a  considerable  investment  in  education 


and  training  is  required.  And,  this  legal  and 
regulatory  environment  is  constantly 
changing,  which  requires  vigilant  atten¬ 
tion  and  incessant  retraining  of  acquisi¬ 
tion  professionals  so  that  their  knowledge 
remains  current. 

Moreover,  few  would  argue  that  cur¬ 
rent  acquisition  processes  perform  in  an 
optimal  manner.  Rather,  the  costs  required 
to  staff  and  manage  defense  procurement 
consume  a  substantial  portion  of  acquisi¬ 
tion  funding,  and  procurement  adminis¬ 
trative  lead  time  (PALT)  represents  a  ma¬ 
jor  factor  in  weapon  system  planning  and 
logistics.  Indeed,  legislation  such  as  the 
Federal  Acquisition  Streamlining  Act 
(FASA)  and  the  Defense  Acquisition 
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Workforce  Improvement  Act  (DAWIA), 
along  with  changes  (e.g.,  performance 
specifications  and  standards,  preference 
for  commercial  items,  integrated  process 
teams)  mandated  from  within  the  Depart¬ 
ment  of  Defense  (DoD),  require  dramatic 
improvement  in  performance  of  the  mili¬ 
tary  procurement  process.  To  exacerbate 
the  difficulties  inherent  in  this  perfor¬ 
mance-improvement  task,  the  mandates 
come  at  a  time  when  defense  appropria¬ 
tions  are  declining  substantially,  acquisi¬ 
tion  staffs  are  getting  much  smaller,  and 
many  procurement  organizations  are  now 
obligated  to  fund  themselves  on  a  fee-for- 
service  basis  (i.e..  Defense  Business  Op¬ 
erations  Fund  (DBOF)). 

The  magnitude  of  required  improve¬ 
ment  is  such  that  “business  as  usual”  no 
longer  represents  a  viable  alternative  for 
acquisition  management.  Neither  can 
management  expect  to  achieve  process 
improvements  of  the  magnitude  required 
through  the  incremental  methods  (Ham¬ 
mer  &  Champy,  1993)  of  continuous  pro¬ 
cess  improvement  (CPI);  rather,  the  kinds 
of  dramatic  performance  improvements 
that  are  needed  call  for  radical  process 
redesign  (Hammer,  1990) — that  is,  the 
military  procurement  process  requires 
reengineering. 


Reengineering 


Business  process  reengineering 
(BPR) — which  is  also  referred  to  as  busi¬ 
ness  engineering  (van  Mael,  1993),  pro¬ 
cess  innovation  (Davenport,  1993),  pro¬ 
cess  redesign  (Davenport  &  Short,  1990), 
and  simply  reengineering  (Hammer, 
1990) — has  been  defined  as  . .  the  fiin- 
damental  rethinking  and  radical  redesign 
of  business  processes  to  achieve  dramatic 
improvements  in  critical,  contemporary 
measures  such  as  cost,  quality,  service,  and 
speed  [emphasis  added]”  (Hammer  & 
Champy,  1993,  p.  32),  The  fundamental 
nature  of  reengineering  relates  to  question¬ 
ing  assumptions;  that  is,  taking  nothing 
about  a  business  or  organization  as  fixed 
or  given,  and  challenging  the  appropriate¬ 
ness  and  existence  of  every  aspect  of  busi¬ 
ness  organization  and  operation.  “Radical” 
redesign  refers  to  transforming  even  the 
most  enduring,  stable,  and  central  aspects 
of  a  design  configuration,  and  envision¬ 
ing  new  redesign  alternatives  without  limi¬ 
tations  or  constraints  associated  with  a 
current  design.  “Dramatic”  improvement 
implies  that  the  level  of  performance  is 
expected  to  increase  by  several  fold  (e.g., 
2x,  5x),  as  opposed  to  incremental  im¬ 
provements  that  are  generally  measured 
in  percentages  (e.g.,  5%,  20%). 


Dr.  Mark  E.  Nissan  is  a  professor  of  systems  management  at  the  Naval  Postgraduate  School 
(NPS),  where  he  teaches  courses  on  information  technoiogy  and  software  acquisition,  and 
conducts  research  on  the  appiication  of  artificial  inteiligence  to  innovate  processes  in  the  ac¬ 
quisition  domain.  After  earning  his  Ph.D.  in  information  systems  from  the  University  of  Southern 
California,  he  taught  MBA  courses  at  the  Haas  School  and  conducted  research  in  electronic 
commerce  at  U.C.  Berkeley  before  joining  the  NPS  faculty  in  July  1996.  Mark  has  over  a  dozen 
years'  aerospace  management  experience  and  was  an  officer  in  the  Supply  Corps  prior  to 
beginning  his  academic  career. 
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Although  the  focus  of  BPR  has  been 
primarily  on  the  private  sector,  many  fed¬ 
eral  agencies  are  also  actively  involved 
with  reengineering.  These  federal 
reengineering  endeavors  generally  fall 
under  the  rubric  of  the  “reinventing  gov¬ 
ernment”  program  (U.S.  Government, 
1993a),  and  have  led  to  the  development 
of  several  “reinvention  labs”  (U.S,  Gov¬ 
ernment,  1993b)  across  the  country.  Nu¬ 
merous  cases  of  reengineering  in  the  gov¬ 
ernment  sector  can  be  found  in  Electronic 
College  (1996).  Although  military  orga¬ 
nizations  and  processes  have  a  number  of 
unique  attributes,  many  of  the  tools,  tech¬ 
nologies,  methods,  and  redesigns  from 
business  process  reengineering  apply 
equally  well  to  organizational  processes 
such  as  military  procurement.  One  such 
currently  emerging  technology  is  knowl¬ 
edge-based  reengineering. 

Khowledge-Based  Ruhgiheerihc 

The  term  knowledge-based  reengi¬ 
neering  refers  to  the  use  of  knowledge- 


based  systems  (KBSs)  to  automate  and 
support  process  redesign.  Integrating  some 
powerful  methods  and  technologies  of  ar¬ 
tificial  intelligence  ( AI)  with  a  number  of 
expert  reengineering  methodologies  from 
the  consulting  practice,  the  knowledge- 
based  system  KOPeR  (knowledge-based 
organizational  process  redesign)  (pro¬ 
nounced  “cope-er”)  was  designed  to  pro¬ 
vide  automated  reengineering  support 
(Nissen,  1995).  The  KOPeR  design  draws 
from  the  AI  methods  and  technologies  that 
led  to  the  development  of  many  powerful 
KBSs,  each  of  which  has  enabled  dramatic 
improvements  in  organizational  processes. 
These  include,  for  example,  the  Rl/XCON 
system  (McDermott,  1982),  MYCIN 
(Shortliffe,  1976),  SOPHIE  (Brown  et  al., 
1982),  the  CLUES  system,  and  the 
Authorizer’s  Assistant  (Laudon  & 
Laudon,  1994,  pp.  599,  606).  Many  such 
“intelligent”  systems  have  demonstrated 
professional  levels  of  performance  that 
meet  and  exceed  those  of  engineers,  phy¬ 
sicians,  and  financial  managers. 

Figure  1  delineates  the  redesign  meth¬ 
odology  supported  by  KOPeR.  An  orga- 
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nizational  process  in  the  field  is  first  rep¬ 
resented  in  terms  of  a  computer-based 
model;  such  computer-based  modeling 
now  represents  standard  reengineering 
practice,  and  the  Integrated  Definition 
family  (e.g.,  IDEF  0,  IDEF  3)  provides  a 
standardized  representational  formalism 
for  modeling  organizational  processes. 
Because  the  IDEF  (and  nearly  all  other 
BPR  modeling)  tools  are  graph-based  (i.e., 
comprised  of  nodes,  edges,  and  attributes), 
a  battery  of  graph-based  diagnostic  pro¬ 
cess  measures 


"KOPeRand  its 
corresponding 
redesign  methodoi- 
ogy  were  used  in  an 
investigation  to 
reengineer  severai 
processes  associated 
with  miiitary  pro¬ 
curement/' 


can  be  obtained 
automatically 
by  KOPeR.  As 
diagnostics, 
these  measure¬ 
ments  are  used 
to  detect  the  se¬ 
vere  patholo¬ 
gies  and  faults 
associated  with 


a  process,  and  KOPeR  employs  its  base 
of  formalized  reengineering  knowledge 
(i.e.,  knowledge  base)  to  predict  which  re¬ 
design  transformations  are  most  likely  to 
effect  dramatic  improvement  in  process 
performance.  These  transformations  are 
then  applied  to  the  baseline  (i.e.,  “as  is”) 
process  model  to  generate  one  or  more  re¬ 
design  alternatives  for  the  process  (i.e.,  the 
“to  be”  scenarios). 

Finally,  once  a  dynamic  process  model 
has  been  validated  and  calibrated  against 
the  process  baseline,  simulation  is  em¬ 
ployed  to  test  the  performance  of  each  re¬ 
design  alternative.  This  represents  a  very 
efficient  technique  to  evaluate  alternate 
process  redesigns,  and  to  reduce  the  in¬ 
herent  risks  of  reengineering  before  com¬ 
mitting  time  and  money  to  a  problematic 
implementation.  Standard  methods  of 


evaluation  and  decision  making  (e.g.,  cost/ 
benefit  analysis,  functional  economic 
analysis,  multi-attribute  decision  making, 
etc.)  can  be  used  to  select  the  highest  po¬ 
tential  redesign  alternative,  which  then  be¬ 
comes  the  focus  of  organizational  plan¬ 
ning  and  implementation. 

Reengineering  the  RFP  Process 

KOPeR  and  its  corresponding  redesign 
methodology  were  used  in  an  investiga¬ 
tion  to  reengineer  several  processes  asso¬ 
ciated  with  military  procurement.  A  ma¬ 
jor  Navy  procurement  organization  was 
the  focus  of  this  investigation,  in  part  be¬ 
cause  its  organization,  processes,  and  tech¬ 
nologies  appear  to  be  representative  of 
those  found  across  any  number  of  mili¬ 
tary  sites  and  bases;  hence  the  results  from 
this  investigation  are  inherently  general- 
izable,  not  only  to  other  Navy  organiza¬ 
tions,  but  to  procurement  processes  across 
the  range  of  service  components,  govern¬ 
ment  agencies,  and  major  corporations. 
This  Navy  procurement  organization  is 
part  of  a  national  reinvention  laboratory, 
which,  by  definition,  has  expressed  an 
explicit  interest  in  process  innovation,  and 
it  is  also  perceived  to  be  well-managed, 
having  already  institutionalized  Total 
Quality  Management  (TQM),  and 
achieved  a  number  of  impressive  process 
improvements  (e.g.,  the  use  of  electronic 
data  interchange  [EDI]  and  bankcards  for 
small  and  micro  purchases,  respectively). 
Because  these  procurement  processes  are 
acknowledged  to  perform  relatively  well 
to  begin  with,  we  have  a  good  opportu¬ 
nity  to  demonstrate  the  use  and  utility  of 
KOPeR  and  knowledge-based  reengi¬ 
neering. 
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Table  1 :  Key  RFP  Process 

Measurements 

Measure 

Value 

Process  size 

255 

Process  length 

255 

Parallelism 

1.00  * 

Handoff  fraction 

0.24 

IT-Support  fraction 

0.08 

IT-Communication  fraction 

0.00 

IT-Automation  fraction 

0.00  * 

*  indicates  theoretical  minimum 

RFP  Process  Measurements 

AND  DUGNOSTICS 


Table  1  summarizes  a  subset  of  the  key 
RFP  process  measurements  and  diagnos¬ 
tics.  The  process  size  (255)  measures  the 
number  of  activities  required  for  an  RFP, 
the  magnitude  of  which  indicates  that  the 
process  is  relatively  large;  process  size  is 
generally  related  positively  to  activity- 
based  cost  (O’ Guinn,  1991),  whereas  pro¬ 
cess  length  is  thought  to  drive  activity 
cycle  time  (Stalk  &  Hout,  1990).  The  par¬ 
allelism  measurement  (size  divided  by 
length)  represents  a  theoretical  minimum 
for  that  measure;  that  is,  the  value  (LOO) 
indicates  that  the  RFP  process  flow  is  en¬ 
tirely  sequential,  with  each  of  its  activi¬ 
ties  performed  in  a  strict  linear  fashion. 
The  handoff  fraction  measures  the  per¬ 
centage  of  process  activities  associated 
with  organizational  handoffs;  handoffs  are 
generally  associated  with  work  sitting  in 
in-boxes  and  out-boxes — awaiting  assign¬ 
ment,  performance,  reviews,  and  manage¬ 
rial  approval — which  adversely  affects 
process  cycle  time.  By  comparison  with 


other  procurement  processes,  the  value 
(0.24)  obtained  for  the  RFP  suggests  that 
much  of  the  PALT  associated  with  this  pro¬ 
cess  may  be  driven  by  handoffs. 

The  fractional  metrics  associated  with 
information  technology  (IT)  measure  the 
density  of  three  classes  of  IT  in  organiza¬ 
tional  processes:  support,  communication, 
and  automation.  The  IT-Support  measure¬ 
ment  (0.08)  is  very  low  for  the  kind  of 
knowledge-  and  information-work  in¬ 
volved  in  the  RFP  process,  and  indicates 
that  most  process  activities  are  performed 
manually.  Worse,  the  IT-Communication 
fraction  is  indistinguishable  from  zero  (to 
2  decimal  places),  which  indicates  that 
nearly  all  process  communications  are 
paper-based.  Moreover,  the  IT-Automa- 
tion  fraction  (0.00)  reflects  the  theoreti¬ 
cal  minimum  for  that  measure,  which  in¬ 
dicates  a  labor-intensive  process. 

To  summarize,  the  baseline  RFP  pro¬ 
cess  represents  a  labor-intensive,  linear 
sequence  of  manual,  paper-based  activi¬ 
ties  that  are  interspersed  between  numer¬ 
ous  handoffs  and  reviews.  The  implica¬ 
tion  is  that  the  process  has  a  number  of 
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pathologies  and  faults,  which  appear  to 
adversely  impact  its  cost  and  cycle  time. 
Recall  that  the  RFP  process  is  currently 
considered  to  represent  a  well-managed 
set  of  acquisition  activities.  Having  diag¬ 
nosed  these  pathologies  and  faults, 
KOPeR  next  concentrates  on  reducing 
process  linearity  through  a  workflow 
transformation,  and  applies  three  IT-based 
redesign  transformations  to  overcome  the 
manual,  paper-based,  labor-intensive 
shortcomings  of  the  process.  KOPeR  em¬ 
ploys  these  transformations  to  generate  the 
redesign  alternatives  below. 

Redesign  Alternatives 


Referring  back  to  the  methodology  de¬ 
picted  in  Figure  1,  KOPeR  utilizes  its  mea¬ 
surements  and  diagnostics  to  predict  one 
or  more  redesign  transformations  that  have 
good  potential  to  effect  dramatic  perfor¬ 
mance  improvement  through  process  in¬ 
novation.  The  methodology  also  calls  for 
a  team  of  process  experts  from  the  pro¬ 
curement  organization  to  review  all  of  the 
KOPeR-generated  transformations,  using 
criteria  such  as  feasibility,  implementabil- 
ity,  perceived  benefit,  and  the  like.  Of  the 
40  or  so  redesign  transformations  pre¬ 
dicted  by  KOPeR,  the  corresponding  mea¬ 
surement  information  helps  the  team  of 
experts  to  focus  on  three  IT-based  enabling 
technologies  that  are  perceived  to  have  the 
highest  potential  for  improvement:  a)  pro¬ 
curement  workflow  systems,  b)  expert  re¬ 
view  systems,  and  c)  knowledge-based 
composition  systems.  As  an  aside,  the 
reader  should  note  that  current 
reengineering  practice  (e.g.,  Davenport, 
1993)  guides  against  process  innovation 
based  solely  on  IT-based  transformations. 


yet  IT  continues  to  represent  the  central 
enabling  technology  for  process  redesign. 
These  transformations  are  discussed  in 
turn. 

Procurement  workflow  systems.  Work- 
flow  systems  can  enable  the  kinds  of  IT- 
based  process  support  and  communication 
that  are  absent  from  the  baseline  RFP  pro¬ 
cess.  Workflow  systems  provide  a  com¬ 
puter-based  infrastmcture  for  the  routing, 
storage,  and  support  of  electronic  docu¬ 
ments,  which  can  greatly  reduce  the  time 
required  for  work  to  move  between  activi¬ 
ties.  The  basic  workflow  system  used  to 
support  the  RFP  process  combines  fea¬ 
tures  generally  associated  with  a  database 
management  system  (DBMS)  for  indexed 
storage  and  retrieval  of  work  documents 
at  various  stages  of  completion,  along  with 
those  of  an  electronic  communication  ap¬ 
plication  such  as  e-mail,  which  can  be  used 
to  transmit  work  documents  to  the  vari¬ 
ous  organizational  agents  involved  in  a 
process.  Additionally,  the  sequence  of 
steps  and  agents  involved  in  a  process  is 
generally  enumerated  beforehand,  and 
used  to  automatically  route  work  to  the 
proper  agent,  when  the  work  is  required 
to  be  completed. 

Other  system  capabilities  can  include 
templates  to  describe  the  overall  flow  of 
work  in  a  process,  along  with  on-line  pro¬ 
cess  “help”  and  reference  information 
(e.g.,  regulations,  contract  clauses,  etc.). 
A  number  of  commercial  workflow  appli¬ 
cations  is  available  on  the  market,  and 
several  firms  have  expertise  in  customiz¬ 
ing  workflow  applications  to  support  a 
particular  organization  and  process  such 
as  Navy  RFP  preparation.  The  implemen¬ 
tation  of  workflow  systems  requires  a  con¬ 
siderable  investment,  however,  not  only 
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in  computer  hardware,  software,  and  de¬ 
velopment,  but  also  in  personnel  training, 
system  maintenance,  and  process  control. 
Interestingly,  at  the  time  of  this  writing, 
the  process  participants  also  indicate  that 
a  decision  has  been  made  to  purchase, 
customize,  and  implement  just  such  a  pro¬ 
curement  workflow  system. 

Review  expert  systems.  Expert  systems 
are  capable  of  performing  many  of  the 
reviews  required  by  contracting  officers 
and  lawyers,  and  can  enable  the  kind  of 
IT-based  automation  of  process  activities 
that  is  missing  from  the  baseline  RFP  pro¬ 
cess.  The  expert  systems  transformation 
addresses  the  various  high-level  reviews 
of  RFP  documents  that  are  conducted  by 
contracting  officers  (KOs)  and  legal  rep¬ 
resentatives.  These  important  reviews  pre¬ 
vent  errors  and  mistakes  from  affecting 
RFPs  and  subsequent  contracting  activi¬ 
ties  downstream  in  the  acquisition  process, 
and  they  require  considerable  skill  and 
experience  on  the  part  of  the  KOs  and  law¬ 
yers  involved.  Hence,  this  requires  rela¬ 
tively  sophisticated  AI  technology  re¬ 
quired  to  reproduce  the  performance  of 
such  highly  skilled  experts. 

Alternatively,  a  great  many  aspects  of 
military  procurement  are  governed  by 
laws  and  regulations,  which  are  articulated 
through  the  U.S.  Code  (USC),  Federal 
Acquisition  Regulations  (FAR),  Defense 
FAR  Supplement  (DFARS),  and  other 
widely  distributed  documentation;  hence, 
many  of  the  RFP  reviews  offer  good  po¬ 
tential  for  automation  via  (esp.  rule-based) 
expert  systems.  Although  complete  auto¬ 
mation  is  impractical  (and  probably  un¬ 
desirable),  the  review  tasks  can  still  be 
shared  between  human  and  computer 
agents.  For  example,  following  something 


of  a  “80/20  rule,”  if  80  percent  of  RFP 
reviews  are  routine  and  perfunctory  in  na¬ 
ture,  then  these  can  be  assigned  to  the 
machine  agent;  the  human  experts  (i.e., 
KOs  and  lawyers)  can  then  handle  the  re¬ 
maining  20  percent  that  are  more  difficult 
or  complex.  Such  a  scheme  for  (human/ 
machine)  task  sharing  has  precedence  in 
industry  (e.g.,  engineering  design,  credit 
review,  loan  approval),  and,  although  its 
transformation  is  predicated  on  relatively 
sophisticated  AI 

"Although  contplefe 
automation  is  im- 
practicai  (and  prob- 
abiy  undesirabie), 
the  review  tasks  can 
stili  be  shared  be¬ 
tween  human  and 
computer  agents." 


technology,  the 
regulation- 
based  nature  of 
the  acquisition 
domain  makes 
it  nearly  ideal 
for  expert  sys¬ 
tems  develop¬ 


ment.  Indeed, 

this  transformation  in  no  way  pushes 
against  the  frontiers  of  AI  in  computer  sci¬ 
ence;  rather,  it  represents  the  adaptation 
of  established  technology  to  the  acquisi¬ 
tion  domain,  and  the  kind  of  expert  RFP 
review  systems  described  here  is  well 
within  current  expert  systems  capabilities. 


Composition  knowledge-based  sys¬ 
tems,  Knowledge-based  systems  are  simi¬ 
lar  in  spirit  to  the  expert  systems  above  in 
that  they  automate  tasks.  The  primary  dif¬ 
ference  lies  in  the  sophistication  of  knowl¬ 
edge-based  systems,  as  they  do  not  require 
performance  at  the  level  of  an  expert  (e.g., 
KO,  lawyer);  rather,  their  level  of  knowl¬ 
edge  and  performance  is  lower,  more  on 
the  order  of  an  analyst  or  information 
worker  like  the  Contract  Specialist  (CS). 
As  such,  the  technological  challenge  is 
greatly  reduced,  even  from  that  of  the  re¬ 
view  expert  systems  above,  yet  the  poten- 
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Table  2:  Redesigned  RFP  Process  Performance 


Redesign  Alternative 

Cost 

Cycletime 

Procurement  Workflow  Systems 

nii 

28% 

Review  Expert  Systems 

11% 

33% 

Composition  KBS 

52%  * 

67%  * 

*  Exceeds  performance-doubling  threshold. 

tial  for  dramatic  performance  improve¬ 
ment  remains.  This  makes  the  KBS  rede¬ 
sign  very  attractive. 

The  work  to  be  accomplished  by  such 
knowledge-based  systems  involves  docu¬ 
ment  composition.  This  generally  entails 
activities  such  as  (a)  reviewing  the  back¬ 
ground  information  and  requirements  as¬ 
sociated  with  a  procurement;  (b)  describ¬ 
ing  the  supplies  and  services  to  be  pro¬ 
cured;  (c)  selecting  the  appropriate 
clauses,  specifications,  and  standards;  (d) 
outlining  proposal  instructions,  schedules, 
and  delivery  information;  and  (e)  using 
these  elements  to  compose  the  proper  RFP 
sections.  Depending  on  the  level  of  de¬ 
sired  “intelligence”  and  functionality,  a 
composition  KBS  can  also  be  developed 
to  make  recommendations  regarding  con¬ 
tract  types,  incentives,  and  other  higher- 
level  activities,  in  addition  to  being  em¬ 
bedded  directly  into  the  kind  of  procure¬ 
ment  workflow  system  described  above. 
Indeed,  in  the  simulations  that  follow,  such 
a  procurement  workflow  system  is  pre¬ 
sumed  to  exist,  and  provides  a  digital 
document  infrastructure  to  support  both 
RFP  composition  and  expert  review.  As 
above,  complete  automation  of  the  RFP 
document  composition  process  is  imprac¬ 


tical  (and  probably  undesirable),  but  ex¬ 
tant  KBS  technology  should  be  capable 
of  incorporating  most  of  the  associated 
process  activities  (perhaps  similar  to  the 
80/20  split  above). 

Simulation  Results _ 

Simulation  represents  a  powerful  com¬ 
puter-based  capability  for  evaluating  the 
relative  performance  of  numerous  alter¬ 
natives  in  a  very  short  amount  of  time,  and 
for  gauging  the  effects  of  management  de¬ 
cision-making  alternatives — such  as  com¬ 
peting  process  redesigns — before  commit¬ 
ting  time  and  money  to  a  particular  alter¬ 
native  (Hansen,  1994;  van  Mael,  1993). 
This  point  is  very  important,  as  many  BPR 
methodologies  omit  this  testing  step,  and 
jump  directly  from  process  redesign  into 
implementation  of  a  particular  technology. 
The  simulation  of  redesign  alternatives  is 
generally  much  less  time-consuming  and 
expensive  than  conducting  pilot  projects, 
particularly  when  a  relatively  large  num¬ 
ber  (say  three  or  more)  of  alternatives  is 
involved;  further,  simulation  is  much, 
much  faster  and  cheaper  than  implement¬ 
ing  an  inappropriate  redesign  alternative. 
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Table  2  provides  a  summary  of  simula¬ 
tion  results  for  the  three  IT-based  RFP  re¬ 
design  alternatives.  The  simulated  perfor¬ 
mance  of  each  IT-based  redesign  alterna¬ 
tive  is  projected  for  a  period  of  one  fiscal 
year,  and  compared  with  that  of  the  RFP 
process  baseline  in  terms  of  activity  cost 
and  cycle  time.  The  figures  shown  in  the 
table  reflect  the  percentage  process  im¬ 
provement  over  the  baseline  RFP  process. 
For  purposes  of  significance,  a  perfor¬ 
mance-doubling  threshold  is  established 
for  “dramatic”  (i.e.,  lx)  improvement;  that 
is,  either  cost  or  cycle  time  must  be  re¬ 
duced  by  at  least  half  (50%)  to  merit  con¬ 
sideration  as  a  process  innovation.  Alter¬ 
natives  that  do  not  to  at  least  double  per¬ 
formance  are  better  classified  as  CPI  ini¬ 
tiatives  than  as  BPR  transformations. 

Workflow  systems  performance.  First, 
notice  that  the  workflow  systems  trans¬ 
formation  effected  only  a  modest  (28%) 
improvement  in  cycle  time,  and  had  a  neg¬ 
ligible  impact  on  process  cost.  Given  the 
potential  for  improvement  noted  above, 
this  may  appear  surprising,  for  workflow 
systems  can  greatly  reduce  the  communi¬ 
cation  time  between  process  activities,  and 
provide  the  capability  to  automatically 
route  work  documents  (at  the  right  time) 
to  the  appropriate  agents.  These  results 
also  ran  counter  to  the  intuition  of  pro¬ 
cess  experts  and  participants,  as  they  con¬ 
sidered  this  technology  to  have  the  great¬ 
est  redesign  potential.  Through  analysis, 
however,  one  discovers  that  workflow 
systems  do  not  change  the  process  activi¬ 
ties  that  must  be  performed;  rather,  they 
change  only  the  environment  and  inter¬ 
face  within  which  the  performance  of  such 
activities  takes  place.  Such  simple  inser¬ 
tion  of  IT  into  an  organizational  process 


has  been  colorfully  referred  to  as  “paving 
the  cowpaths”  and  “automating  the  mess” 
(Hammer,  1990).  Using  a  workflow  inter¬ 
face  can  sometimes  even  impede  the  per¬ 
formance  of  certain  tasks. 

For  example,  using  a  slow,  tedious,  or 
non-intuitive  interface  to  consult  on-line 
regulations  or  other  references  can  require 
more  effort  than  simply  opening  a  (paper- 
based)  desk  ref- 

erence  (e.g.,  the  "Such  simple  inser- 
FAR).  Addi-  lion  of  IT  Ini®  Bin 

tionally,  most  organixalionol 
1  n  process  has  beeet 

workflow  sys  referred  I® 

terns  are  config-  „.p^i„glhe 
ured  with  the  ,owpnlhs'  and  'a«l®- 
electronic  moling  the  mess'" 

equivalents  of  (Hammer,  1990) 

in-boxes  (e.g., 
task  lists)  and 

out-boxes,  as  agents  must  deliberately 
“open”  their  tasks  before  processing. 
Therefore,  although  the  communication 
between  process  activities  is  much  faster 
through  workflow  systems,  without  rede¬ 
signing  the  underlying  process  itself,  the 
actual  latency  of  documents  awaiting  pro¬ 
cessing  does  not  necessarily  improve 
through  the  technology.  Moreover,  imple¬ 
mentation  of  IT  such  as  a  procurement 
workflow  system  requires  customization, 
training,  maintenance,  and  support,  in  ad¬ 
dition  to  the  associated  computer  and  net¬ 
work  hardware  and  software  costs.  Based 
on  these  results,  the  procurement  work- 
flow  system  does  not  portend  the  kind  of 
dramatic  performance  improvement  ex¬ 
pected  through  process  innovation — at 
least  not  as  a  standalone  alternative,  or  as 
an  end  in  and  of  itself. 

Expert  systems  performance.  The  re¬ 
view  expert  systems  redesign  reduced 
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both  cost  (11%)  and  cycle  time  (33%)  for 
the  RFP  process.  Notice,  however,  that 
neither  the  cost  nor  cycle-time  improve¬ 
ment  exceeds  the  performance-doubling 
threshold.  These  results  also  appear  sur¬ 
prising,  and  ran  counter  to  the  intuition  of 

process  experts 

"...the  knowledge- 
based  systems  used  psnts;  they  con¬ 
fer  RFP  document  sidered  this  ad- 

compesition  ex-  vanced  technol- 

ceeded  the  perfer-  ogy  to  have  ex- 

itiance  doubling  cellent  redesign 

threshold  lor  both  potential,  and, 

cost  (52%)  and  cycle  gi^^n  the  fact 

time  <«7%)...."  jjjg  k;o  and 

legal  review 
tasks  are  automated  through  the  expert 
systems,  much  greater  improvements  in 
process  performance  were  expected. 

Through  analysis  one  discovers  that, 
although  these  review  tasks  represent  criti¬ 
cal  steps  to  ensure  the  quality  and  profes¬ 
sionalism  of  RFPs,  the  reviews  do  not  rep¬ 
resent  high-leverage  activities  for  process 
redesign;  in  other  words,  the  review  ac¬ 
tivities  are  very  important,  but,  as  a  per¬ 
centage  of  the  total  process  cost  and  cycle 
time,  they  do  not  represent  the  major  con¬ 
tributors  in  terms  of  activities.  Hence,  only 
modest  performance  improvements  can  be 
expected  to  accrue  through  the  automa¬ 
tion  of  these  RFP  review  activities. 

Knowledge-based  systems  perfor¬ 
mance,  In  contrast  with  the  modest  per¬ 
formance  improvements  projected  above 
for  the  workflow-  and  expert-systems  re¬ 
design  transformations,  the  knowledge- 
based  systems  used  for  RFP  document 
composition  exceeded  the  performance¬ 
doubling  threshold  for  both  cost  (52%) 
and  cycle  time  (67%),  reflecting  the  kind 


of  dramatic  gains  expected  through 
reengineering.  These  results  were  also  sur¬ 
prising  and  counter-intuitive,  but  for  very 
different  reasons.  As  noted  above,  the 
composition  KBS  transformation  repre¬ 
sents  less  technological  sophistication 
with  respect  to  the  review  expert  systems, 
yet  the  payoff  from  this  redesign  interven¬ 
tion  is  dramatic.  This  provides  an  inter¬ 
esting  insight  into  the  relationship  between 
technological  sophistication  (and  risk)  and 
process  improvement  (i.e.,  payoff),  and 
casts  some  doubt  on  a  widely  espoused 
notion  that  major  improvements  require 
the  most  advanced  technologies. 

Through  analysis  one  discovers  that  the 
relative  success  of  this  redesign  transfor¬ 
mation  is  attributable  to  the  high  leverage 
associated  with  RFP  composition  activi¬ 
ties.  Recall  that  this  provides  a  contrast  to 
the  relatively  low  leverage  observed 
through  the  RFP  review  activities  from 
above.  Specifically,  unlike  the  RFP  review 
activities  that  are  automated  through  the 
expert  systems  transformation — which  we 
indicated  contributes  a  relatively  low  per¬ 
centage  to  total  process  cost  and  cycle 
time — the  RFP  composition  activities  con¬ 
stitute  the  bulk  of  process  tasks  required 
to  prepare  RFPs.  In  other  words,  for  this 
particular  RFP  process,  the  contract 
specialist’s  document-composition  activi¬ 
ties  offer  the  greatest  potential  for  im¬ 
provement  in  terms  of  automation,  and  the 
KBS  technology  targets  these  composition 
activities  directly. 

Finally,  it  is  important  to  reiterate  that 
all  three  of  these  IT-based  transformations 
neglect  the  integration  of  other,  well-es¬ 
tablished  enablers  of  process  innovation 
(e.g.,  organizational  design,  and  human  re¬ 
sources;  see  Davenport,  1993).  Although 
a  number  of  corresponding,  non-IT  trans- 
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formations  (e.g.,  case  manager,  process 
teams,  delegation,  empowerment,  incen¬ 
tives)  were  identified  by  KOPeR,  the  team 
of  process  experts  did  not  feel  as  though 
changes  to  the  existing  procurement  or¬ 
ganizational  or  human-resource  structure 
would  be  feasible  at  the  time  of  the  inves¬ 
tigation.  As  noted  above,  concentrating 
solely  on  IT-based  transformations  is  not 
recommended  by  current  reengineering 
practice.  However,  considering  that  the  IT- 
based  transformations  above  may  still  be 
combined  with  other  (i.e.,  non-IT)  enablers 
at  a  later  date,  the  near-term  prospect  for 
effecting  the  magnitude  of  process  im¬ 
provement  examined  through  this  inves¬ 
tigation  must  be  exciting  to  today’s  pro¬ 
curement  manager.  Have  faith,  because 
current  practice  also  recommends  against 
trying  to  do  everything  at  once. 

Conclusions  and  Future  Research 


This  article  has  investigated  the  rede¬ 
sign  of  the  RFP  process  as  conducted  by 
one  well-managed  Navy  procurement  or¬ 
ganization.  The  research  shows  that 
KOPeR  proved  to  be  successful  in  sup¬ 
porting  the  modeling,  measurement,  diag¬ 
nosis,  and  redesign  of  the  RFP  process, 
and  simulation  played  a  critical  role  in  the 
evaluation  and  comparison  of  redesign  al¬ 
ternatives.  Without  the  simulation  results 
above,  for  example,  other  procurement 
managers  may  have  been  tempted  to 
implement  procurement  workflow  sys¬ 
tems  similar  to  the  transformation  de¬ 
scribed  above,  without  an  appreciation  for 
the  expected  costs  and  limitations  of  this 
redesign  alternative.  Based  solely  on  the 
procurement  workflow  system,  such  man¬ 
agers  can  anticipate  negligible  overall  cost 


savings,  and  only  modest  improvement  in 
cycle-time  performance;  hence,  the  pru¬ 
dence  of  this  redesign  decision  (i.e.,  to 
implement  only  the  workflow  system) 
appears  to  be  questionable.  This  result 
may  provide  useful  guidance  to  the  de¬ 
signers  and  decision  makers  now  respon¬ 
sible  for  the  DoD  Standard  Procurement 
System  (SPS)  development  (SPS,  1996); 
the  functionality  of  SPS  is  expected  to  be 
very  similar  to  the  procurement  workflow 
systems  described  above. 

This  research  also  identified  the  review 
activities  of  contracting  officers  and  legal 
advisors  as  tar¬ 
get  opportuni-  |fBS  transfor- 

ties  for  automa-  mafioii  exceeded  all 
tion  through  ex-  ether  redesign  alter- 
pert  systems,  natives  in  terms  of 
KOs  and  law-  dramatic  gerfermance 

yers  represent  *™P>'wement." 

well-paid, 

highly  trained  professionals  that  are  criti¬ 
cal  to  the  RFP  process,  and  automating 
their  reviews  through  expert-systems  tech¬ 
nology  was  perceived  to  represent  an  ex¬ 
cellent  opportunity  for  process  improve¬ 
ment,  However,  even  with  the  workflow 
systems  infrastructure  (which  KOPeR  in¬ 
dicated  was  a  necessary  condition  for  ex¬ 
pert  systems’  efficacy),  this  redesign  trans¬ 
formation  failed  to  exceed  the  perfor¬ 
mance-doubling  threshold  for  process  im¬ 
provement.  Particularly  during  times  of 
tight  fiscal  constraint,  it  may  be  prudent 
to  subordinate  this  redesign  transforma¬ 
tion  to  others  that  reflect  greater  potential 
for  dramatic  performance  improvement. 

Specifically,  the  document-composition 
that  KBS  succeeded  in  more  than  halving 
both  the  cost  and  cycle  time  of  the  RFP  pro¬ 
cess.  Even  though  it  uses  less-sophisticated 
AI  technology  than  the  review  expert  sys- 
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terns,  the  KBS  transformation  exceeded  all 
other  redesign  alternatives  in  terms  of  dra¬ 
matic  performance  improvement.  This  high¬ 
lights  an  exciting  opportunity  to  further  ex¬ 
plore  such  composition  KBSs,  and  possi¬ 
bly  to  couple  this  promising  KBS  technol¬ 
ogy  with  current  efforts  to  develop  and 
implement  the  next  generation  of  procure¬ 
ment  workflow  systems  (e.g.,  SPS). 

Further,  a  system  such  as  KOPeR  may 
itself  complement  current  efforts  to  cap¬ 
ture  other  acquisition  processes  through 
computer-based  modeling.  Because  of 
KOPeR’ s  measurement-driven  diagnostic 
capability,  it  has  the  potential  to  run  “in 
the  background”  (i.e.,  as  a  non-user-di- 
rected  process)  of  emerging  process  tools 
such  as  KnowledgeWorker  (Knowledge- 

Worker,  1996), 


"Finally,  acquisition 
does  not  represent  a 
solitary  activity; 
rather,  it  involves 
the  interaction 
between  a  procuring 
organization  and 
one  or  more  contrac¬ 
tors  submitting 
proposals  and  sup¬ 
plying  goods  and 
services." 


serving  to  sug¬ 
gest  potential 
redesign  trans¬ 
formations  even 
as  the  basic  pro- 
cess  informa¬ 
tion  is  being 
captured  and 
formalized 
through  the  sys¬ 
tem.  Indeed,  be¬ 
cause  workflow 


systems  themselves  depend  on  a  (com¬ 
puter-based)  process  model  to  function, 
^KOPeR  could  also  operate  “behind  the 
scenes”  (i.e.,  outside  of  the  users’  imme¬ 
diate  views),  diagnosing  pathologies  and 
faults  in  the  underlying  workflow-en¬ 
hanced  processes  (e.g.,  helping  to  avoid 
“paving  the  cowpaths”  and  “automating 
the  messes”).  This  represents  another  in¬ 
teresting  area  for  future  research. 

As  an  immediate  topic  of  related  re¬ 
search,  recall  that  this  present  investiga¬ 


tion  is  designed  to  produce  results  that  are 
broadly  generalizable.  This  suggests  that 
KOPeR  and  the  RFP  redesign  transforma¬ 
tions  described  in  this  paper  offer  good 
potential  to  innovate  the  RFP  and  like  ac¬ 
quisition  processes  in  other  organizations. 
Given  the  regulatory  nature  of  defense 
acquisition,  one  would  expect  to  see  con¬ 
siderable  similarity  among  the  procure¬ 
ment  processes  of  various  naval  com¬ 
mands  (e.g.,  NAVAIR,  NAVSEA),  service 
components  (e.g,.  Air  Force,  Army),  gov¬ 
ernment  agencies  (e.g.,  DOE,  DOT),  and 
major  corporations.  Generalizing  the  re¬ 
sults  of  this  present  investigation  repre¬ 
sents  an  opportunity  to  leverage  the  ben¬ 
efits  of  KOPeR  and  knowledge-based 
reengineering,  which  could  help  spark  AI- 
based  acquisition  process  innovation  on  a 
wide  scale. 

Finally,  acquisition  does  not  represent 
a  solitary  activity;  rather,  it  involves  the 
interaction  between  a  procuring  organiza¬ 
tion  and  one  or  more  contractors  submit¬ 
ting  proposals  and  supplying  goods  and 
services.  This  interaction  suggests  that  the 
kind  of  research  described  in  this  paper 
can  be  extended,  for  example  through  in¬ 
vestigations  to  integrate  the  govemment- 
RFP  and  contractor-proposal  processes. 
Such  integration  could  lead  to  the  discov¬ 
ery  of  additional  opportunities  for  dra¬ 
matic  performance  improvement  through 
process  redesign.  In  the  spirit  of  acquisi¬ 
tion  reform  and  integrated  process  teams, 
research  along  this  line  also  has  the  po¬ 
tential  to  reduce  contractor  costs  (and 
cycle  times)  as  well  as  those  incurred  by 
procuring  organizations  directly.  Clearly, 
much  work  remains  to  be  done,  but 
KOPeR  and  its  IT-based  process  innova¬ 
tions  appear  to  offer  great  potential. 
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LESSONS  LEARNED 


DEVEIOPMENT  OF  EMERGING 
TELEMEDICINE  TECHNOLOGIES  WITHIN 
THE  DEPARTMENT  OF  DEFENSE: 

A  CASE  STUDY  OF  OPERATION  JOINT 
ENDEAVOR  IN  BOSNIA 


Conrad  A,  Clyburn,  CaryR.  Gilbert,  Timothy  J.  Cramer, 
Roy  K.  Lea,  Stott  G.  Ehnes,  and  Russ  Zaitthuk 

For  the  past  several  years  the  United  States  Department  of  Defense  has  been 
using  rapid  prototyping  as  well  as  integrated  product  and  process  management 
to  enhance  military  health  care  delivery  using  new  telemedicine  technologies. 
This  paper  outlines  how  these  concepts  are  being  used  within  the  DoD 
Telemedicine  Testbed  to  conduct  proof-of-principle  demonstrations  and 
operational  testing  of  emerging  telemedicine  technologies  within  real-world 
military  operational  environments.  The  authors  consider  these  rapid  prototyping 
and  “rapid  integration”  efforts  to  be  advanced  development  phases  of  the 
Army’s  formal  telemedicine  acquisition  program  known  as  Medical 
Communications  for  Combat  Casualty  Care  and  DoD’s  Theater  Medical 
Information  Program.  They  examine  a  case  study  of  a  real-time  application  of 
rapid  prototyping  in  a  contemporary  operation— Primetime  III,  the  telemedicine 
augmentation  of  medical  units  supporting  Operation  Joint  Endeavor  in  Bosnia. 
The  lessons  learned  from  this  experience  are  important  to  the  eventual 
achievement  of  the  strategic  objectives  of  DoD  telemedicine. 


During  the  past  two  years  the  Secre¬ 
tary  of  Defense  has  articulated  a 
strategy  for  a  new  system  for  Depart¬ 
ment  of  Defense  (DoD)  research,  develop¬ 
ment  and  acquisition  that  shifts  from  tradi¬ 
tional  product-based  research  and  develop¬ 
ment  to  a  dynamic  new  framework  charac¬ 
terized  by  rapid  prototyping,  concurrent  as 


opposed  to  sequential  engineering,  reliance 
on  commercial-off-the-shelf  (COTS)  tech¬ 
nologies,  horizontally  and  vertically  inte¬ 
grated  teamwork,  and  experimentation  as 
the  heart  of  integrated  product  and  process 
management  (DoD,  1996). 

The  DoD  Telemedicine  Test  Bed  has 
adopted  these  principles  in  its  own  inte- 
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grated  framework  for  telemedicine  re¬ 
search,  development,  and  acquisition.  In 
this  system  the  Defense  Advanced  Re¬ 
search  Projects  Agency  (DARPA)  in  part¬ 
nership  with  the  Medical  Research  and 
Materiel  Command  (MRMC)  performs 
basic  research  and  develops  new  technolo¬ 
gies,  such  as  telerobotic  surgery,  the  per¬ 
sonnel  status  physiological  monitor, 
trauma  and  clinical  care  decision  support 
systems,  surgical  simulators,  and  ad¬ 
vanced  diagnostic  imaging  systems.  At  its 
Prototype  and  Implementation  Test  Lab 
(PITLAB),  the  Army  Medical  Research 
and  Materiel  Command’s  Medical  Ad¬ 
vanced  Technology  Management  Office 
(MATMO)  integrates  new  technology  de¬ 
veloped  by  DARPA  and  COTS  technolo¬ 
gies  into  prototype  systems  tailored  to  sat¬ 
isfy  military  requirements  as  articulated 


by  the  Army,  Navy,  and  Air  Force  user 
communities  (Figure  1). 

Prototypes  are  then  evaluated  by  users 
in  a  variety  of  settings.  These  settings 
range  from  Advanced  Warfighting  Experi¬ 
ments  (AWEs)  staged  at  U.S.  Army  Battle 
Labs,  or  joint  service  exercises  in  which 
the  Army,  Navy  and  Air  Force  use  test 
environments  appropriate  to  their  unique 
missions.  Another  option  for  blended  tele¬ 
communications  and  advanced  medical 
diagnostic  technologies  is  the  Center  for 
Total  Access,  a  joint  initiative  between  the 
U.S.  Army  Signal  Center  and  the  Dwight 
David  Eisenhower  Army  Medical  Center 
at  Fort  Gordon,  Georgia.  Large  field  medi¬ 
cal  units  at  Fort  Hood,  Texas,  and  Fort 
Bragg,  North  Carolina,  also  offer  excel¬ 
lent  test  beds  for  user  experimentation  and 
feedback. 
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Figure  1«  Department  of  Defense  Telemedicine,  Research, 
Development,  and  Acquisition 


In  addition,  tertiary  care  regional  mili¬ 
tary  medical  centers  such  as  the  Walter 
Reed  Army  Medical  Center  and  Bethesda 
Naval  Medical  Center,  both  in  Washing¬ 
ton,  D.C,,  Wilford  Hall  Air  Force  Medi¬ 
cal  Center  in  Texas,  Tripler  Army  Medi¬ 
cal  Center  in  Hawaii,  and  Madigan  Army 
Medical  Center  in  the  state  of  Washing¬ 
ton  have  been  active  test  sites  for  both 
peacetime  and  combat  support  telemed¬ 
icine  evaluation  programs  (Crowther  and 
Poroputich,  1995;  Lyche  et  al.,  1995; 
Delaplain,  Linborg,  Norton,  and  Hastings, 
1993).  Contemporary  operations  such  as 
the  U.S.  Armed  Forces  commitments  to 
Somalia,  Haiti,  the  Middle  East,  or  Op¬ 
eration  Joint  Endeavor  in  Bosnia  have  of¬ 
fered  a  rich  environment  for  rapid 
prototyping  of  integrated  technology  test¬ 
ing  (Cawthon  et  al.,  1991;  Detreville  et 


al.,  1995;  Gomez,  1994;  Crowther  and 
Poropatich,  1995),  because  they  constitute 
a  full  range  of  health  care  services  with 
networked  telecommunications  and  infor¬ 
mation  technology  implementations  rang¬ 
ing  from  far  forward  combat  casualty  care 
applications  to  subspecialty  teleconsul¬ 
tation  at  the  tertiary  care  medical  centers. 
The  DoD  Telemedicine  Testbed  at  Fort 
Detrick  (MATMO)  also  leverages  tele¬ 
medicine  experience  and  work  done  out¬ 
side  the  DoD  (Sanders  and  Bashshur, 
1995;  Galvin  et  al,  1995;  Goldberg  et  al., 
1994;  Perednia  and  Allen,  1995;  Mun, 
Alsayed,  Tohme,  and  Wu,  1995).. 

Within  this  rapid  prototyping  model, 
validated  technologies  destined  for  com¬ 
bat  casualty  care  support  are  transitioned 
to  a  Program  Manager  (PM).  The  Army 
has  recently  established  a  PM  for  Medi- 
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cal  Communications  for  Combat  Casualty 
Care  (MC4).  The  PM  is  responsible  for 
acquisition  of  a  completed  technology 
system.  He  ensures  that  effective  “process 
management”  strategies  are  in  place  and 
that  appropriate  organizational,  concept, 
doctrine,  and  training  issues  have  been 
addressed  prior  to  large-scale  acquisition 
and  fielding  to  operational  forces. 

The  introduction  of  robust  telecommu¬ 
nications,  information  technology  and 
advanced  diagnostic  capabilities  into 

health  care  has 
profound  ef¬ 
fects  on  estab¬ 
lished  work  pat- 
terns  and  re¬ 
sponsibilities. 
Successful 
technology  in- 
t  e  g  r  a  t  i  o  n 
projects  require 
comprehensive 
user  require¬ 
ments,  organi¬ 
zational  work  process,  and  training  analy¬ 
sis  prior  to  large-scale  acquisition  and  to¬ 
tal  package  fielding  (Appleton,  1993).  The 
PM  works  closely  with  the  user  commu¬ 
nities  to  ensure  that  important  organiza¬ 
tional  assumptions  and  strategies  regard¬ 
ing  human-computer  interfaces,  work  pat¬ 
tern  changes,  and  personnel  management 
are  tested  before  there  is  widespread  field¬ 
ing. 

The  DoD  can  take  advantage  of  a  wide 
range  of  assets  to  further  develop 
telemedicine.  These  assets  are  coupled 
with  the  DoD’s  relative  immunity  from  a 
number  of  complex  private  sector  impedi¬ 
ments  to  telemedicine  development.  These 
impediments  include  restricted  interstate 
medical  licensing,  lack  of  fee-for-service 


"The  introduction  of 
robust  telecommuni¬ 
cations,  information 
technology  and 
advanced  diagnostic 
capabilities  into 
health  care  has 
profound  effects  on 
established  work 
patterns  and  respon¬ 
sibilities*" 


reimbursement,  and  unclear  liability  rela¬ 
tionships  (Perednia  and  Allen,  1 993).  These 
circumstances  place  DoD  in  a  unique  posi¬ 
tion  to  explore,  evaluate,  and  continuously 
improve  telemedicine  technologies  across 
a  broad  spectmm  of  clinical  applications  and 
user  settings.  This  exploration  could  ben¬ 
efit  both  soldiers  and  civilians. 


Medical  Communications  for  Combat 
Casuaity  Care  (MC4) 


The  U.S.  Army  formally  articulates  its 
program  for  telemedicine  support  of 
ground  forces  as  having  five  strategic 
thrusts.  Far  forward  applications  will  pro¬ 
vide  front-line  medics  and  small  forward 
medical  support  units  with  still  imagery, 
interactive  videoteleconferencing  (VTC), 
and  diagnostic  scopes.  Eventually,  far  for¬ 
ward  medics  will  have  access  to  technol¬ 
ogy  that  can  monitor  the  physiological  sta¬ 
tus  of  individual  soldiers  and  provide  real 
time  medical  situational  awareness  of 
where  and  how  medical  assets  are  being 
used  on  the  battlefield. 

The  Digital  Field  Medical  Treatment 
Facility  will  provide  ground  support  forces 
with  telemedicine-augmented  deployable 
hospitals.  These  hospitals  will  include 
capabilities  for  computed  radiography, 
computed  tomography,  teleradiology 
transmission,  VTC  and  diagnostic  scopes, 
high-resolution  still  imagery,  teledentistry, 
computerized  hospital  and  patient  infor¬ 
mation  systems,  Internet  access,  and  en¬ 
hanced  wireless,  hospital,  and  patient  in¬ 
formation  management  systems. 

The  Expert  Tertiary  Care  Host  provides 
forward  deployed  forces  with  access  to 
specialty  and  subspecialty  teleconsultation 
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from  medical  centers  located  in  the  conti¬ 
nental  United  States.  These  requirements 
include  the  development  of  technology  for 
regional  medical  centers  to  send  and  re¬ 
ceive  transmissions  from  remotely  located 
forward  deployed  medical  units,  and  to 
distribute  VTC,  image  acquisition,  and 
electronic  patient  information  throughout 
the  hospital.  Telemedicine  capabilities 
within  medical  centers  will  be  based  on 
the  concept  of  clinical  workstations  that 
provide  the  clinician  with  an  integrated 
telepresence  that  includes  real  time  VTC, 
ubiquitous  access  to  digital  medical  im¬ 
aging,  and  patient  information. 

Telemedicine  Sustainment  ensures  that 
a  strategic  telecommunications  infrastruc¬ 
ture  is  available  to  support  telemedicine 
requirements  from  the  continental  United 
States  to  forward  deployed  locations 
around  the  world.  The  telemedicine  sus¬ 
tainment  thrust  includes  ensuring  the 
availability  of  high-bandwidth  satellite 
and  terrestrial  communications  to  de¬ 
ployed  medical  forces  and  research  into 
the  enhanced  utilization  of  frame  relay  and 
asynchronous  transfer  mode  (ATM)  tech¬ 
nologies  to  support  multimedia,  simulta¬ 
neous  medical  transmissions  of  voice,  text, 
and  video.  Another  important  area  of  con¬ 
cern  is  the  development  of  enhanced  medi¬ 
cal  compression  algorithms  that  will  al¬ 
low  the  transmission  of  medically  relevant 
information  at  lower  bandwidths. 

The  last  area,  Emerging  Technologies, 
supports  the  integration  of  what  is  now 
basic  research  into  prototyping,  advanced 
development,  and  eventually,  fielding  to 
operational  forces.  These  technologies 
include  the  further  development  of  ad¬ 
vanced  surgical  simulators  to  enhance  sur¬ 
gical  training  and  planning;  the  integra¬ 
tion  of  the  personnel  status  physiological 


monitor  into  the  broader  U.S.  Army  War¬ 
rior  Status  monitor  program;  and  the  test¬ 
ing,  evaluation,  and  fielding  of  the  Life 
Support  for  Trauma  and  Transport 
(LSTAT)  system,  a  self-contained  critical 
care  transport  module  with  noninvasive, 
miniaturized  diagnostic  modalities,  resus¬ 
citation,  information  management,  and 
communications  capabilities.  The  contin¬ 
ued  research  into  surgical  robotics  and 
expert  decision  support  systems  is  another 
area  of  research  interest  that  DARPA  has 
funded  for  development. 

Theater  Medical  Information 
Program  (TMIP) 


The  Theater  Medical  Information  Pro¬ 
gram  (TMIP)  is  DoD’s  programmatic  ef¬ 
fort  to  fully  integrate  telemedicine  and 
medical  informatics  systems  support  for 
deployed  forces  from  all  services.  The 
concept  is  to  provide  a  seamless  continuum 
of  information-based  health  care  services 
from  the  most  forward  deployed  forces 
around  the  world  to  the  sustaining  base  DoD 
and  Veteran’s  Affairs  medical  centers  in  the 
United  States.  To  accomplish  this,  the  TMIP 
will  integrate  and  extend  support  to  de¬ 
ployed  forces  for  DoD’s  functional  medical 
information  systems  such  as  the  Composite 
Health  Care  System  (CHCS)  (originally  de¬ 
signed  as  a  peace-time  inpatient  and  outpa¬ 
tient  hospital  information  system),  the  De¬ 
fense  Blood  Standard  System  (DBSS),  the 
Transportation  Command  Regulating  and 
Command  &  Control  Evacuation  System 
(TRAC2ES)  and  the  Theater  Army  Medi¬ 
cal  Management  Information  System 
(TAMMIS).  The  Army’s  MC4  program  is 
an  integral  part  of  the  TMIP  effort. 
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Context  For  Primetime  III  - 
Telemedicine  Augmentation  to  Bosnia 

When  President  Bill  Clinton  directed 
the  deployment  of  U.S.  Armed  Forces  in 
support  of  the  Dayton  Peace  Accords  in 
mid-December  of  1995,  one  of  the  pri¬ 
mary  concerns  of  U.S,  policy  makers  was 
the  possibility  of  incurring  casualties 
among  U.S.  ground  forces.  This  concern 
was  credible  in  the  aftermath  of  U.S.  ca¬ 
sualties  in  Somalia,  of  the  shooting  down 
of  a  U.S.  aircraft  in  the  months  prior  to 
the  Dayton  Peace  Accords,  and  numerous 
hostage  takings  (and  releases)  of  United 
Nations  soldiers  earlier  in  1995. 

Various  combinations  of  telemedicine 
technologies  have  been  tested  in  joint  ser¬ 
vice  exercises,  U.S.  Army  AWEs,  on  board 
deployed  naval  vessels,  and  within  the 
peacetime  Military  Health  Service  System 
(MHSS).  The  U.S.  military  has  also  ex¬ 
ecuted  real  world  missions  with  these  tech¬ 
nologies  in  a  wide  variety  of  contingency 
operations  over  the  previous  five  years. 

Deployments  in  which  telemedicine  has 
been  used  to  provide  real  world  medical 
support  include  Saudi  Arabia,  Kuwait,  So¬ 
malia,  Haiti,  Cuba,  Panama,  Croatia,  and 
Macedonia.  By  the  time  of  the  Dayton 
Peace  Accords,  continued  improvements 
in  commercially  available  information 
technologies  such  as  computer  network¬ 
ing,  data  compression,  and  high-band¬ 
width  telecommunications,  linked  with 
improved  medical  imaging  and  diagnos¬ 
tics  technologies,  offered  military  medi¬ 
cal  planners  with  an  unprecedented  oppor¬ 
tunity  to  deploy  integrated  telemedicine 
capabilities  far  forward  into  a  military  the¬ 
ater  of  operations. 

Consequently,  the  senior  civilian  and 
military  leadership  of  the  Military  Health 


Service  System  decided  in  late  December 
of  1995  to  use  a  wide  variety  of  telemedi¬ 
cine  technologies  to  augment  the  U.S. 
medical  units  assigned  to  Task  Force 
Eagle,  the  U.S.  component  of  Operation 
Joint  Endeavor  in  Bosnia  and  Hungary,  as 
supported  from  the  Landstuhl  Regional 
Medical  Center  in  Germany.  The  techni¬ 
cal  support  task  force  formed  to  imple¬ 
ment  this  intent  is  called  Primetime  III. 
The  Commander  of  U.S.  Army  Medical 
Research  and  Materiel  Command  was 
delegated  direct  responsibility  for  the  day- 
to-day  supervision  and  execution  of  the 
mission. 


Primetime  III 


Primetime  III  is  a  real-world  rapid 
prototyping  proof  of  principle  demonstra¬ 
tion  for  the  MC4  and  TMIP  programs  de¬ 
scribed  above;  this  is  a  “first-shot”  effort 
at  deploying  an  integrated  theater  medi¬ 
cal  information  solution  tailored  to  the 
specific  missions  of  Task  Force  Eagle  and 
Operation  Joint  Endeavor. 

The  telemedicine  solutions  deployed 
are  intended  to  allow  medical  command¬ 
ers  to  satisfy  five  user  defined  require¬ 
ments:  a)  to  provide  total  patient  account¬ 
ability  and  medical  regulating  (tracking 
and  scheduling  patients  through  the  evacu¬ 
ation  chain)  in  real  time  from  the  first  phy¬ 
sician  encounter;  b)  to  keep  evacuations 
and  patient  movement  to  a  minimum  and 
maximize  return  to  duty;  c)  to  deliver  rapid 
world-class  response  to  trauma;  d)  to  le¬ 
verage  the  expertise  of  medical  special¬ 
ists  located  in  Germany  and  the  United 
States  in  support  of  forward  deployed  units 
in  Bosnia  and  Hungary;  and,  e)  to  provide 
medical  leadership  with  enhanced  situ- 
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ational  awareness  through  networked  in¬ 
formation  management  systems  and  VTC 
at  all  levels  of  medical  support. 

Deployed  Technologies _ 

Primetime  III  has  provided  Operation 
Joint  Endeavor  medical  units  with  the 
most  robust  set  of  telemedicine  technolo¬ 
gies  deployed  in  a  contemporary  opera¬ 
tion  to  date.  Primetime  III  was  planned 
and  executed  using  rapid  prototyping  and 
“rapid  systems  integration”  strategies;  the 
Primetime  III  Task  Force  director  em¬ 
ployed  a  “brokering”  strategy,  bringing  a 
variety  of  systems  prototypes  and 
prototypers  together  to  produce  a  loosely 
integrated,  nearly  complete  systems  solu¬ 
tion  to  the  telemedicine  and  medical 
informatics  objectives.  This  horizontally 
and  vertically  matrixed  approach  mini¬ 
mized  organizational  mass  while  optimiz¬ 
ing  function  by  leveraging  the  capabili¬ 
ties  of  DoD,  Army,  Navy,  and  Air  Force 
units,  the  Defense  Information  Systems 
Agency  (DISA),  DARPA,  and  industry- 
academic  partners  including  GTE, 
Northrop  Grumman,  Teleos,  N.E.T., 
SAIC,  United  Medical  Network,  United 
Medical  Network,  Johnson  &  Johnson 
(Ethicon  Division),  CLI,  Picturetel,  Vec¬ 
tor  Research,  and  Georgetown  University. 

Units  were  fielded  with  Picturetel  and 
CLI,  Inc.,  high-resolution,  384-kilobyte- 
per-second  (kbs),  30  frame-per-second 
(fps)  systems  capable  of  high  bandwidth, 
real  time  interactive  VTC.  American 
Medical  Development  provided  integrated 
digital  diagnostic  scopes  sets  to  augment 
the  interactive  VTC. 

Georgetown  University  designed,  de¬ 
ployed,  and  installed  a  radiology  local  area 


network  (LAN)  system  capable  of  provid¬ 
ing  filmless  medical  imaging  and  trans¬ 
mission  of  computed  tomography  (CT) 
scanning,  computed  radiography  (CR), 
and  full-motion,  color  doppler  ultrasound. 

United  Medical  Network  developed  a 
deployable  telesurgical  mentoring  operat¬ 
ing  room  suite  that  provides  laparoscopic 
scope,  overhead,  and  room  view  interac¬ 
tive,  bi-directional  audio  and  video  trans¬ 
mission  to  support  complicated  surgical 
cases  in  Bosnia.  A  similar  digital  radiol¬ 
ogy  and  teleradiology  system  was  em¬ 
ployed  onboard  aircraft  carriers  offshore. 
This  configuration  of  deployed  hospitals, 
medical  centers  in  Germany  and  in  the 
United  States, 

and  shipboard  «p,|melliiie  III  has 
hospitals,  each  provided  Operation 

configured  with  Joint  Endeavor 
an  internal  radi-  medical  units  with 
ology  LAN  in-  the  most  robust  set 
terconnected  to  of  telemedicine 
the  other  hospi-  technologies  de- 
tals  by  a  virtual  •••  *•  contem- 

wide  area  net  »• 

work  (WAN),  is 
the  first  radiol¬ 
ogy  implementation  of  the  seamless  con¬ 
tinuum  of  support  envisioned  by  the 
DoD’s  TMIP  program. 

The  U.S.  Air  Force  MEDSITE  team 
from  Brookes  Air  Force  Base  in  San  An¬ 
tonio,  Texas,  in  conjunction  with  the  Army 
office  for  Defense  Medical  Information 
Systems  at  Walter  Reed  Army  Medical 
Center  in  Washington,  D.C.,  provided  and 
fielded  an  enhanced  deployable  version  of 
the  CHCS,  the  DoD’s  integrated  hospital 
information  system.  As  deployed, 
CHCS(D)  initiates  computerized  medical 
record  keeping  in  the  field;  facilitates  far 
forward  delivery  of  laboratory,  radiologi- 
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cal,  and  prescription  support;  and  provides 
clinical  electronic  mail  for  collaborative 
care  among  widely  dispersed  clinicians. 
In  another  entrepreneurial  effort,  patient 
administration  and  medical  regulating 
functionality  of  the  Army’s  Theater  Army 
Medical  Information  System,  the 
CHCS(D),  and  the  Defense  Medical 
Regulating  Information  System  (DMRIS) 
were  significantly  enhanced  by  a  proto¬ 
type  PC/Intemet-based  real-time  patient 
tracking  system  developed  by  the  Army’s 
Patient  Accounting  Systems  and  Biostatis- 
tical  Agency  (PASBA),  Dubbed  PARRTS 
(Patient  Accounting  and  Real-time  Re¬ 
porting  Transmission  Systems),  this  pro¬ 
totype  system  allows  military  medical  of¬ 
ficers  and  tactical  commanders  to  access 
computerized  patient  data  and  track 
evacuations.  Primetime  III  also  provides 
enabling  technology  to  access  online  clini¬ 
cal  databases  via  the  military  and  civilian 


internet  and  to  deliver  teledentistry  services. 

The  Uniformed  Services  University  of 
Health  Sciences  Combat  Casualty  Care 
Research  Center  provided  all  on-site  clini¬ 
cal  training.  The  DoD  Health  Affairs 
TMIP  office,  in  conjunction  with  D ARPA, 
is  providing  the  medical  informatics  inte¬ 
gration  strategy. 

All  these  systems  are  supported  by  a 
collaborative  MATMO,  DIS A,  GTE  tele¬ 
communications  systems  integration  so¬ 
lution  that  provides  satellite  and  terrestrial 
high  bandwidth  (approximately  1 .536  Mb/ 
s  (megabits  per  second))  telecommunica¬ 
tions  support  to  forward  forces  at  Taszar, 
Hungary,  and  Tuzla,  Bosnia,  128  kbs  Inter¬ 
national  Maritime  Satellite  (INMARSAT) 
communications  to  dispersed  clinics 
throughout  Bosnia,  and  384  kbs  terrestrial 
connectivity  to  U.S.  military  medical  cen¬ 
ters  in  Washington,  D.C.,  Texas,  Califor¬ 
nia  and  Hawaii. 


PRIMETIME  III  CAPABILITIES  LIST 

FILMLESS,  COMPUTED 

Satellite 

CLINICAL  E-MAIL 

RADIOGRAPHY 

T-1 

1.536  Mb/s 

TELERADIOLOGY  GATEWAYS 

COMPUTED  TOMOGRAPHY 

Terrestrial 

FULL-MOTION  DOPPLER 

HIGH-RESOLUTION 

T-1 

ULTRASOUND 

VIDEOCONSULTATION 

1.536  Mb/s 

LAPRASCOPIC  &  ROOM  VIEW 

DIGITAL  DIAGNOSTICS 

384  kbs 

TELESURGICAL  MENTORING 

30  fps 

STILL  IMAGING 

Interactive 

TELEDENTISTRY 

VTC 

HOSPITAL  &  PATIENT 

INTERNET  ACCESS 

INFORMATION  SYSTEMS 

TCP/IP 

Medical  Image 

Transfer 

MEDICAL  SITUATIONAL  AWARENESS 

Figure  2.  Primetime  III  Telemedicine  and  Medical  Informatics 

Capabilities  List 


108 


Development  of  Emerging  Telemedicine  Technologies  Within  the  Department  of  Defense 


A  combination  of  satellite  and  terres¬ 
trial  telecommunications  allows  for  con¬ 
nectivity  with  the  USS  George  Washing¬ 
ton  and  USS  Enterprise  aircraft  carriers 
offshore  in  the  Adriatic  Sea.  The  carriers 
were  equipped  with  similar  capabilities  for 
filmless  radiology,  teleradiology  transmis¬ 
sion,  real-time,  interactive  VTC,  and  pa¬ 
tient  information  systems  (Figure  2). 

CoMCEPT  OF  Operation 


Primetime  III  is  an  example  of  a  mis¬ 
sion-critical  new  technology  integration 
project  executed  under  intensive  deadline 
pressure.  Less  than  two  months  elapsed 
from  the  time  the  operation  was  ordered 
to  the  time  the  first  augmentation  was 
fielded  at  Landstuhl  Regional  Medical 
Center  (LRMC). 

The  operation  was  originally  organized 
into  three  telemedicine  augmentation 
phases.  A  fourth  phase  was  added  to  fo¬ 
cus  on  the  integration  of  emerging  tech¬ 
nologies,  including  the  development  of  a 
transmittable  and  transportable  electronic 
patient  record.  Other  potential  technology 
introductions  include  a  prototype  three- 
dimensional  portable  ultrasound,  remote 
PC-based  telepathology,  and  the  LSTAT 
system. 

In  Phase  One,  the  task  force  fielded 
telemedicine  augmentation  packages  and 
supporting  high-bandwidth  communica¬ 
tions  to  the  LRMC  in  Germany,  the  67th 
CSH  in  Hungary  and  the  212th  Mobile 
Army  Surgical  Hospital  (MASH)  in 
Bosnia.  The  LRMC  is  a  fixed  facility  re¬ 
gional  center  servicing  active  duty  and 
family  members  in  Germany;  the  67th 
CSH  and  212th  MASH  are  mobile  army 
hospitals  deployed  in  the  operational  area. 


consisting  of  a  combination  of  tents  and 
hardened  shelters.  In  addition,  during 
Phase  One,  the  USS  George  Washington 
was  equipped  and  positioned  off  the  coast 
of  Bosnia  in  the  Adriatic  Sea.  High-band- 
width  telecommunications  and  VTC  were 
also  provided  to  military  community  hos¬ 
pitals  and  command  and  control  elements 
in  Germany. 

In  Phase  Two,  six  operating  base  medi¬ 
cal  units  in  Bosnia  were  fielded  and 
trained.  These  medical  units  are  much 
smaller  than  the  hospitals  and  are  designed 
to  support  troop  concentrations  of  a  few 
thousand.  They  have  a  limited  holding 
capacity  for  short  periods  of  time,  and  re¬ 
ceived  lower  bandwidth,  more  easily 
deployable  INMARSATS  (128  kbs)  sup¬ 
ported  capabilities  for  VTC,  still  imaging, 
medical  information  system  access,  and 
at  some  sites,  teleradiology  transfer. 

Phase  Three  is  intended  to  provide 
telemedicine  coverage  for  the  remaining 
operating  base  medical  units.  At  this  writ¬ 
ing  the  exact  number  and  location  of  these 
units  is  being  determined  by  theater  medi¬ 
cal  commanders  based  on  projected  op¬ 
erational  requirements  in  the  later  stages 
of  Operation  Joint  Endeavor.  The  intent 
throughout  the  operation  has  been  to  pro¬ 
vide  total  package  fielding,  including  com¬ 
prehensive  new  equipment  training,  main¬ 
tenance  and  sustainment  of  telemedicine 
augmentation  packages.  Task  force  per¬ 
sonnel  have  made  multiple  trips  to  Europe 
for  forward  fielding,  installation,  training, 
and  follow-up. 

The  completed  augmentation  is  an  in¬ 
tegrated,  worldwide  telemedicine-enabled 
health  care  delivery  system  (Figure  3)  ex¬ 
tending  from  remote  clinic  sites  in  Bosnia 
to  major  medical  centers  in  the  United 
States. 
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Figure  3.  Functional  and  Operational  Concept  for  Primetime  III 
Telemedicine  Support  for  Operation  Joint  Endeavor 
in  Bosnia  and  Hungary 


The  medical  centers  designated  to  pro¬ 
vide  direct  support  to  forces  in  Bosnia  are 
organized  to  be  able  to  provide  24-hour 
time  zone  arrayed  teleconsultation  support 
(Figure  4).  One  medical  center  (Walter 
Reed  Army  Medical  Center)  is  located  on 
the  east  coast  of  the  United  States.  The 
second  medical  center  (Wilford  Hall  Air 
Force  Medical  Center)  is  located  in  the 
center  of  the  United  States  in  another  time 
zone.  The  Navy  Medical  Center  in  San 
Diego  is  located  in  California,  andTripler 
Army  Medical  Center  is  located  in  another 
time  zone  in  the  Pacific  Ocean.  Each  hos¬ 
pital  has  a  slightly  different  composition 
of  specialist  and  core  competencies.  The 
combination  of  time  zones  and  specialty 
populations  offers,  clinicians  in  Bosnia 
with  a  deep  reservoir  of  teleconsultation 


support,  continuing  medical  education, 
and  medical  information. 

System  Utilization 


Initial  results  from  the  first  four  months  of 
the  operation  are  shown  in  Tables  1  and  2. 
Table  1  displays  the  recorded  system  use 
and  Table  2  depicts  the  operational  reli¬ 
ability  (up-time)  of  the  various  technologies 
and  functionalities  employed.  Radiology 
was  the  most  widely  used  telemedicine  ap¬ 
plication.  Since  the  only  radiologist  present 
within  the  theater  of  operations  was  lo¬ 
cated  at  the  CSH,  physicians  at  the  MASH 
had  to  use  teleradiology  for  radiology  pro¬ 
cedures  that  required  the  presence  of  a  ra¬ 
diologist.  More  than  800  X-rays  were 
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PRIMETIME  III  TASK  FORCE 

PROVIDING  24-HOUR  T-MED  COVERAGE  TO  BOSNIA 


TIME  OF  DAY: 

6  12  15 

BOSNIA,  HUNGARY  oaytime 

LANDSTUHL  :  .  MvnME 

5/6 

WALTER  REED 
WILFORD  HALL  l 

SAN  DIEGO 


18 


24 


17/18 


DAYTIME 

DAYTIME 


18 


DAYTIME 


18 


TRIPLER 


DAYTIME 


Figure  4.  Medical  Center  Telemedicine  Consultation  Support  for 
Operation  Joint  Endeavor,  Organized  to  Provide  24-ltour  Coverage 


transmitted  from  the  MASH  in  Bosnia  to 
a  CSH  in  Hungary  during  the  first  four 
months  of  the  operation.  Similar  use  was 
made  of  the  CT  and  ultrasound  applica¬ 
tions.  Still  image  and  full  motion  video 
consultation  capabilities  were  used  less. 


Several  factors  may  account  for  these  re¬ 
sults.  First,  the  caseload  in  Bosnia  and 
Hungary  during  this  period  was  very  light. 
The  troops  deployed  were  in  generally 
good  health  and  there  was  no  significant 
hostile  action.  Second,  the  training  and  op- 


Table  1*  System  Utilization 


APPLICATION 

OP  BASES  TO 
LRMC 

MASH/CSH  TO 
LRMC/CONUS 

67th  CSH  TO 
LRMC/CONUS 

NAVY 

CARRIERS  TO 
CONUS 

Clinical  VTC 

4 

28 

39 

36 

Digital  Radiography 

N/A 

3,969 

Transfer  814 

2,413 

3,700  T-RAD 

CT  Scanner 

N/A 

207 

37 

Not  Reported 

Uitrasounds 

N/A 

98 

Not  Reported 

8 

Continuing  Medical  Education 
(CME) 

0 

40 

3 

2 
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Table  2*  System  Operational  Reiiabiiily  (%  Up-Time) 


SYSTEM 

OP  BASES 

MASH/CSH 

67th  CSH 

NAVY 

CARRIERS 

Clinical  VTC 

No  Data  to  Date 

97% 

100% 

100% 

High  Resolution  Still  Image 

No  Data  to  Date 

100% 

N/A 

N/A 

Computed  Radiography 

No  Data  to  Date 

97% 

97% 

100% 

CT  Scanner 

N/A 

95% 

95% 

N/A 

Ultrasounds 

N/A 

99% 

98% 

No  Data  to  Date 

Laparoscopic/Telesurgery 

N/A 

75% 

N/A 

N/A 

Teledentistry 

No  Data  to  Date 

N/A 

100% 

N/A 

Medical  Information  Systems 

No  Data  to  Date 

100% 

100% 

100% 

Continuing  Medical  Education 

No  Data  to  Date 

70% 

75% 

No  Data  to  Date 

erational  support  for  the  systems  was  ini¬ 
tially  inadequate.  Third,  the  systems  ex¬ 
perienced  significant  end-to-end  down¬ 
time  due  to  insufficient  technical  support, 
slow  delivery  of  repair  parts,  and  initial 
inadequate  network  operational  priority. 
In  the  case  of  high-resolution  still  imag¬ 
ery  at  the  MASH,  the  capability  was  in¬ 
advertently  omitted  from  Phase  1  imple¬ 
mentation  all  together.  (See  communica¬ 
tions  and  sustainment  discussions  below.) 
These  shortcomings  were  subsequently 
addressed  by  the  task  force  director  but 
are  strongly  reflected  in  the  initial  system 
utilization  results.  The  most  significant 
uses  of  the  full-motion  VTC  capability  were 
for  graduate  medical  education  (GME)  and 
career  counseling.  The  laparoscopic  sur¬ 
gery  technique  was  highly  accepted  by  the 
physicians  in  theater  but  due  to  lack  of 
caseload  there  were  very  few  surgeries  con¬ 
ducted.  The  teledentistry  system  was  also 
popular  and  easy  to  use  but  was  not  imple¬ 
mented  until  Phase  2,  and  therefore  no  ini¬ 
tial  results  were  available  at  this  writing. 


In  the  area  of  medical  informatics,  the 
success  of  the  deployable  CHCS  hospital 
information  system  within  the  deployed 
hospitals  was  apparent  from  the  start.  It 
was  professionally  installed  and  ad¬ 
equately  supported  by  the  Air  Force 
MEDSITE  team;  training  provided  was 
excellent  and  most  of  the  health  care  pro¬ 
viders  had  had  previous  experience  with 
the  system  at  their  permanent  duty  sta¬ 
tions,  so  user  satisfaction  was  high.  Since 
the  forward  areas  order  entry  and  results 
retrieval  application  of  CHCS  was  not 
implemented  until  Phase  2,  no  initial  re¬ 
sults  were  available  at  this  writing. 

Like  CHCS,  the  prototype  PARRTS 
system  was  highly  accepted  by  users  both 
in  the  area  of  operations  and  in  the  sup¬ 
porting  medical  centers  and  commands. 
Even  the  service  surgeons  general  and 
Assistant  Secretary  of  Defense  for  Health 
Affairs  in  Washington,  D.C.,  logged  on  to 
the  PARRTS  system  via  the  Internet  to  ob¬ 
tain  daily  Operation  Joint  Endeavor  medi¬ 
cal  situation  updates.  A  total  of  2,357  pa- 
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tients  were  registered  in  the  system  (i.e., 
treated  at  the  MASH  or  the  CSH)  within 
the  first  six  months  of  the  overall  opera¬ 
tion. 


Initial  Lessons  Learned _ 

Space  Communications  and 
Network  Management 

The  Space  Communications  and  Net¬ 
work  Management  Team  was  responsible 
for  the  technical  design  and  integration  of 
the  telemedicine  network.  This  team  cre¬ 
ated  the  detailed  communications  and  for¬ 
ward  deployed  networking  lay-down  that 
provided  for  a  commercially  available, 
standards-based  and  scaleable  infrastruc¬ 
ture  capable  of  supporting  time-sensitive 
applications.  The  network  was  to  have  a 
multiuser,  multifunctional  architecture 
that  maximizes  infrastructure  reuse.  The 
network  was  to  require  minimal 
customization  with  maximized  use  of 
COTS  equipment  for  rapid  procurement, 
validation,  and  deployment.  The  network 
was  also  to  have  a  minimum  need  for  on¬ 
site  communications  staffing  and  a  remote 
network  management  capability. 

Within  the  time,  cost,  and  operational 
constraints  of  Operation  Joint  Endeavor, 
all  of  these  objectives  were  unobtainable. 
Rather  than  build  a  separate  telecommu¬ 
nications  network  optimized  to  meet  the 
aforementioned  medical  requirements,  the 
task  force  shared  bandwidth  on  the  com¬ 
mon  high-bandwidth  telecommunications 
network  provided  in  theater.  This  network 
was  not  optimized  for  medical  use,  and 
was  unavailable  at  times.  Because  it  was 
not  a  dedicated  medical  network,  medical 
personnel  did  not  always  have  adequate 
visibility  of  the  causes  nor  the  final  solu¬ 


tions  to  network  telecommunications 
problems  that  affected  the  continuity  of 
telemedicine  enabled  care. 

A  “stovepiped”  medical  telecommuni¬ 
cations  network,  restricted  to  medical  traf¬ 
fic  and  fully  under  the  control  of  medical 
personnel,  would  probably  have  provided 
optimal  functionality;  medical  personnel 
could  have  rapidly  isolated,  identified,  and 
corrected  network  problems.  In  practice, 
this  approach  is  not  cost  effective,  it  ef¬ 
fectively  inhibits  information  interchange 
between  medical  and  nonmedical  com¬ 
manders,  and  is  not  operationally  feasible 
from  warship  medical  facilities.  Shared 
telecommunications  among  medical  and 
nonmedical  requirements  in  deployed  en¬ 
vironments  is  the  most  feasible  use  of 
scarce  bandwidth  in  the  austere  opera¬ 
tional  environments  most  likely  to  require 
telemedicine  support.  In  the  future,  medi¬ 
cal  personnel 
will  have  to  bet¬ 
ter  communi¬ 
cate  their  re¬ 
quirements, 
telecommuni¬ 
cations  provid¬ 
ers  will  have  to 
better  under¬ 
stand  how 
medical  re¬ 
quirements  dif¬ 
fer  from  other 
forms  of  com¬ 
munication,  and 

better  tools  must  be  made  available  to 
medical  personnel  to  monitor  the  perfor¬ 
mance  of  shared  telecommunications 
pipelines.  In  this  way,  medical  personnel 
will  be  able  to  anticipate  problems  and 
bottlenecks  and  proactively  take  correc¬ 
tive  action  to  ensure  that  long-range  com- 


"Shared  telecemmu- 
nications  among 
medical  and  non¬ 
medical  require¬ 
ments  in  deployed 
environments  is  the 
most  feasible  use  of 
scarce  bandwidth  in 
the  austere  opera¬ 
tional  environments 
most  likely  to  re¬ 
quire  telemedicine 
support*" 
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munications  are  available  based  on  medi¬ 
cal  need. 

A  corollary  to  this  finding  is  that  medi¬ 
cal  organizations  need  to  restructure  to 
provide  the  expertise  necessary  to  per¬ 
form,  or  assist  in  the  function  of  network 
conceptualization,  development,  and  man¬ 
agement  in  future  operations.  Even  if  mili¬ 
tary  medical  personnel  had  obtained  the 
full  remote  monitoring,  diagnostic,  and 
network  management  capability  that  was 
sought,  it  is  not  altogether  clear  where 
DoD  medical  personnel  would  have  se¬ 
cured  sufficient 
skilled  person¬ 
nel  to  perform 
remote  network 
management 
and  trouble¬ 
shooting  24 
hours  a  day,  7 
days  a  week. 
The  DISA  has 


telemedicine 
networks  are  estab¬ 
lished  foremost  to 
support  the  physi¬ 
cians  that  must 
deliver  health  care 
in  an  austere  and 
demanding  environ¬ 
ment." 


traditionally 
provided  long- 
haul  strategic  communications,  but  if  the 
agency  is  to  effectively  support  functional 
technologies  such  as  telemedicine,  provid¬ 
ing  a  reliable  back-bone  communications 
infrastructure  is  not  enough.  More  flex¬ 
ible  and  responsive  network  management 
and  prioritization  policies  and  procedures 
as  well  as  end-to-end  systems  integration 
services  and  support  are  also  essential.  As 
health  care  delivery  moves  further  into  the 
integration  of  telecommunications  and 
medicine,  it  is  important  that  we  think  cre¬ 
atively  about  changed  roles  and  responsi¬ 
bilities.  One  option  is  to  leverage  existing 
communications  capabilities,  such  as  call 
centers,  in  major  medical  centers. 


Clinical  Operations  and  Training 

From  the  outset,  the  Clinical  Operations 
and  Training  were  viewed  as  a  key  suc¬ 
cess  factor  of  this  rapid  prototyping  imple¬ 
mentation.  The  Clinical  Operations  and 
Training  Team  was  responsible  for  coor¬ 
dinating  with  and  supporting  the  medical 
commanders  in  developing  a  comprehen¬ 
sive  telemedicine  program. 

Telemedicine  changes  the  familiar  pat¬ 
terns  of  health  care  delivery  and  referral. 
These  new  patterns  must  be  anticipated, 
trained  for,  and  used  in  order  to  achieve 
desired  clinical  outcomes.  The  telemedi¬ 
cine  networks  are  established  foremost  to 
support  the  physicians  that  must  deliver 
health  care  in  an  austere  and  demanding 
environment.  The  clinical  team  is  the  phy¬ 
sician  advocate  within  the  Primetime  III 
task  force. 

The  clinical  team  consulted  widely  in 
theater  and  at  supporting  medical  centers 
to  develop  guidance  on  a  wide  range  of 
clinical  issues.  Among  these  were  the 
clinical  utilization  protocols  for  particu¬ 
lar  telemedicine  technologies,  medical 
center  teleconsultation  and  scheduling 
policy,  appropriate  documentation  for 
telemedicine  enabled  procedures,  imple¬ 
mentation  of  a  technology  acceptance  and 
utilization  strategy,  and  identifying  con¬ 
tinuing  medical  education  (CME)  use  of 
the  network. 

The  clinical  team  developed  training 
plans  and  a  training  schedule  that  aligned 
with  the  projected  phased  site  implemen¬ 
tation  time  line.  The  training  addressed  not 
only  the  operation  of  the  equipment,  but 
also  the  full  spectrum  of  protocol  taxono¬ 
mies  and  telemedicine  applications  for  the 
provision  of  health  care. 

In  practice,  we  found  that  the  effective¬ 
ness  of  training  varied  between  units.  In 
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some  cases,  the  task  force  had  to  redeploy 
trainers  to  provide  additional  sustainment 
training.  This  area — training  development 
and  execution — requires  additional  focus. 
Human-technology  interface,  work  pattern 
changes,  and  medical  efficacy  concerns 
that  many  clinicians  believe  telemedicine 
raises  are  specific  areas  that  require  addi¬ 
tional  research  and  special  attention.  Over¬ 
coming  these  human  issues  are  the  most 
difficult  element  of  most  government  re¬ 
lated  organizational-technological  im¬ 
provement  programs  (Candle,  1994). 

A  second  difficult  area  is  coordinating 
interactions  between  U.S.  medical  centers 
and  the  forward-deployed  field  hospitals. 
There  are  many  opportunities  for  interac¬ 
tion  between  forward  facilities  and  medi¬ 
cal  centers.  In  practice,  the  level  of  inter¬ 
action  between  the  U.S.  medical  centers 
and  the  units  in  theater  varied  widely. 
Some  medical  centers  experienced  fairly 
frequent  contact,  others  rare  contact.  One 
of  the  impediments  to  greater  contact  may 
have  been  that  we  lacked  concrete  infor¬ 
mation  and  procedures  for  interaction  be¬ 
tween  the  two  levels  of  care.  An  infrastruc¬ 
ture  of  procedures  and  detailed  resources 
available  at  each  medical  center  and  field 
treatment  facility  would  probably  facili¬ 
tate  more  effective  and  frequent  consulta¬ 
tion.  This  too  is  an  area  that  requires  ad¬ 
ditional  research. 

At  a  minimum,  training  of  clinicians  on 
telemedicine  must  start  in  the  peacetime 
health  care  delivery  system  and  be  sus¬ 
tained  in  routine  medical  practice,  if  it  is 
to  be  optimized  in  deployed  settings.  Stan¬ 
dardized  training  materials,  not  only  in  the 
operation  of  telemedicine  equipment,  but 
also  in  the  process  changes  that 
telemedicine  creates,  must  be  developed. 
These  new  procedures  must  be  used,  and 


refined  in  routine  medical  practice  and 
institutional  training  programs. 

Though  the  task  force  devoted  signifi¬ 
cant  resources  to  the  identification  and 


resolution  of  technical  problems,  we  may 
have  underestimated  the  difficulty  of  man- 
aging  human 
and  organiza¬ 
tional  change  in 
a  compressed 
time  frame  un¬ 
der  stressful 
conditions. 

These  process 
changes  are  es¬ 
sential  if  these 
new  technolo¬ 
gies  are  to  be 
used  to  their  full 
advantage  in 

enhancing  access  to  and  quality  of  care. 
Maneuvering  full  circle  to  the  emerging 
concept  of  the  Secretary  of  Defense  for 
DoD  Research,  Development  and  Acqui¬ 
sition,  we  must  devote  equal  resources  to 
both  “product”  and  “process”  change  in 
an  integrated  approach. 


"In  its  early  stages, 
a  rapid  pretotyping 
development  model 
is  not  intended  to  be 
a  complete  solution; 
therefore  opera¬ 
tional  sustainment  is 
not  likely  to  hove 
been  fully  planned 
and  tested  prior  to 
prototype  testing  by 
the  user*" 


Operations  and  Sustainment 

Like  training,  operational  sustainment 
and  logistical  support  are  as  key  to  the 
success  of  rapid  prototyping  efforts  as  they 
are  to  traditional  development,  acquisition, 
and  fielding  processes,  especially  if 
prototyping  is  to  occur  in  real-world  situ¬ 
ations.  The  major  weakness  of  the 
Primetime  III  prototyping  effort  is  the  lack 
of  integrated  logistical  support.  Under  tra¬ 
ditional  product  acquisition  methodolo¬ 
gies,  a  complete  operational  and  logisti¬ 
cal  sustainment  package  is  developed  and 
tested  as  part  of  the  acquisition  process. 
In  its  early  stages,  a  rapid  prototyping  de- 
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velopment  model  is  not  intended  to  be  a 
complete  solution;  therefore  operational 
sustainment  is  not  likely  to  have  been  fully 
planned  and  tested  prior  to  prototype  test¬ 
ing  by  the  user.  But  if  this  user  testing  is 
intended  to  be  carried  out  in  a  real-world 
environment,  the  prototyper  must  either 
provide  “hand-holding”  operational  and 
logistical  support  or  develop  a  sustainment 
package. 

In  the  case  of  Primetime  III,  direct  lo¬ 
gistical  support  and  indirect  operational 
support  was  intended  by  the  task  force 
director.  This  proved  to  be  insufficient,  if 
not  unworkable.  Providing  technical  prob¬ 
lem  analysis,  maintenance  troubleshoot¬ 
ing,  repair  parts,  and  component  replace¬ 
ment  support  from  MATMO  headquarters 
in  Frederick,  Maryland,  even  through  a 
network  of  government  and  contractor 
support  personnel  in-theater,  did  not  work 
effectively.  Repair  parts  and  component 
replacements  could  not  be  procured  within 
the  United  States  and  shipped  into  the  the¬ 
ater  of  operations  fast  enough  to  be  re¬ 
sponsive  to  the  medical  task  force  needs. 
Likewise,  user-operator  confidence  and 
proficiency  languished  each  time  the  clini¬ 
cal  operations  and  training  team  withdrew; 
when  user  technical  proficiency  lan¬ 
guished  so  did  use  of  the  system  for  con¬ 
sultations.  To  remedy  these  problems,  the 
Primetime  III  task  force  inserted  perma¬ 
nent  technical  and  operational  support 
personnel  on  the  ground  to  help  person¬ 
nel  use  the  system,  conduct  troubleshoot¬ 
ing  and  system  repairs,  and  coordinate 
assistance  from  MATMO  and  the  Prime¬ 
time  III  contractors. 

Clearly,  rapid  prototyping  operations 
that  are  conducted  in  real-world  environ¬ 
ments,  especially  in  remote  ones,  must  be 
directly  supported  by  the  prototyper  on¬ 


site  or  must  be  modified  to  provide  ad¬ 
equate  user  operational  training  and  an 
integrated  logistical  sustainment  frame¬ 
work  prior  to  deployment.  MATMO  and 
the  U.S.  Army  Medical  Command  are 
currently  developing  plans  for  a  provi¬ 
sional  Deployable  Telemedicine  Units  at 
Fort  Bragg,  North  Carolina,  and  Fort 
Hood,  Texas,  as  an  organic  component  of 
the  Army’s  operational  Forces  Command 
44th  Medical  Brigade.  The  intention  is  to 
improve  medical  readiness  and  the  effec¬ 
tiveness  of  future  telemedicine  deploy¬ 
ments  by  fielding  prototype  telemedicine 
systems  directly  to  operational  forces  for 
operational  testing,  training,  and  develop¬ 
ment  of  logistic  support  plans  and  proce¬ 
dures  prior  to  deployment. 

Evaluation 

No  problem  solution  methodology  is 
effective  without  an  effective  means  of 
evaluating  the  results.  The  most  widely 
accepted  scientific  approach  to  evaluation 
is  a  prospective  approach  with  control 
groups.  In  the  case  of  Primetime  III  the 
evaluation  effort  focused  on  operational, 
clinical,  and  technical  aspects  of  the  op¬ 
eration.  The  effort  was  intended  to  be  pro¬ 
spective  from  the  beginning;  significant 
monetary,  personnel,  and  materiel  re¬ 
sources  were  devoted  toward  evaluation. 
This  included  hiring  two  independent  con¬ 
tractor  organizations  to  support  the 
military’s  own  evaluation  effort:  one  to 
evaluate  the  technical  feasibility  and  op¬ 
erational  performance  of  the  Primetime  III 
systems;  the  other  (hired  by  the  DoD 
Telemedicine  Evaluation  Working  Group 
[TWEG])  to  support  the  Army  Medical 
Department  (AMEDD)  Test  Board  in  its 
own  efforts  to  evaluate  the  operational  and 
clinical  aspects  of  the  operation.  The 
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AMEDD  Test  Board  actually  sent  a  team 
of  military  data  collectors  into  the  theater 
to  observe  the  clinical  use  and  operational 
performance  of  the  Primetime  III 
telemedicine  systems,  as  well  as  the 
Primetime  III  task  force  team  itself.  A  Lo¬ 
tus  Notes  database  designed  to  capture  key 
evaluation  data  elements  was  developed 
and  fielded  as  part  of  the  evaluation  ef¬ 
fort.  The  evaluation  contractors  relied 
heavily  on  data  collected  by  the  AMEDD 
Test  Board  team  via  the  Lotus  Notes  sys¬ 
tem  and  on  the  contents  of  clinical  and 
administrative  e-mail  to  prepare  their 
evaluation  reports. 

The  evaluation  team  is  responsible  for 
ensuring  that  the  in-theater  data  collection 
process,  which  will  form  the  basis  for 
evaluation  analyses,  is  seamlessly  inte¬ 
grated  into  the  health  care  delivery  pro¬ 
cess,  and  does  not  impede  the  health  ser¬ 
vice  mission.  Questions  that  the  combined 
clinical  and  technical  evaluation  will  ad¬ 
dress  include  the  following:  Does  tele¬ 
medicine  reduce  the  anticipated  level  of 
patient  evacuations?  Does  it  reduce  turn¬ 
around  time  of  radiological  and  laboratory 
reports? 

Does  telemedicine  reduce  the  travel 
distance  and  time  required  to  receive  spe¬ 
cialty  care  and  consultation?  Does  it  re¬ 
duce  patient  admissions  and  increase  the 
percentage  of  return  to  duty?  Does  it  re¬ 
duce  cost  of  radiological  film  processing? 
Does  filmless  radiology  reduce  logistical 
requirements?  Does  telemedicine  provide 
better  patient  imaging  than  current  film- 
based  processes?  Does  telemedicine  re¬ 
duce  the  loss  of  medical  images  (films)? 
Does  it  decrease  the  retake  rate  of  radio¬ 
logical  images,  and  does  it  reduce  supply 
and  personnel  costs  versus  the  develop¬ 
ment  of  film?  Does  telemedicine  increase 


the  efficiency  and  effectiveness  of  medi¬ 
cal  staff?  Under  what  conditions  is  tele¬ 
medicine  most  useful? 

A  related,  but  separate  line  of  inquiry 
are  questions  of  technical  adequacy.  For 
example,  does  the  technology  adequately 
support  the  clinical  needs  and  processes? 
Are  the  components  of  the  telemedicine 
augmentation  sets  interoperable?  What 
factors  affect  the  input  of  medical  data? 
What  factors  affect  the  throughput  of 
medical  data?  What  factors  affect  data 
compression  and  pretransmission  image 
modification? 

What  factors  af¬ 
fect  image  reso¬ 
lution  and  video 
display  effec¬ 
tiveness?  How 
does  the  tech¬ 
nology  effect 
the  efficiency 
and  effective¬ 
ness  of  the  de¬ 
livery  of  health 
care? 

While  no  de¬ 
finitive  evalua¬ 
tion  reports 

have  yet  been  submitted,  early  observa¬ 
tions  by  the  evaluation  teams  were  most 
useful  to  the  Primetime  III  task  force  di¬ 
rector  in  initiating  technical,  operational, 
and  clinical  improvements  in  the  system 
and  to  the  support  structure. 

Some  initial  system  utilization  and 
performance  data  are  shown  in  Tables 
1  and  2,  It  is  clear  from  this  data  that 
first,  there  is  insufficient  data  for  a  ro¬ 
bust  evaluation  of  the  clinical  effective¬ 
ness  of  the  telemedicine  system — pri¬ 
marily  because  of  the  lack  of  patient 
workload  in  theater. 


''The  evaluation 
team  is  responsible 
for  ensuring  that  the 
in-theater  data 
collection  process, 
which  will  form  the 
basis  for  evaluation 
analyses,  is  seam¬ 
lessly  integrated 
into  the  health  care 
delivery  process,  and 
does  not  impede  the 
health  service  mis¬ 
sion/' 
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The  troops  deployed  were  basically 
healthy  and  there  was  virtually  no  hostile 
activity  during  the  period  of  data  collec¬ 
tion.  Therefore  any  evaluation  of  the  use¬ 
fulness  of  the  telemedicine  system  for  the¬ 
ater  combat  casualty  care  based  on  the  data 
collected  will  be  biased  due  to  lack  of 
caseload. 

Second,  while  the  system  technical  and 
operational  reliability  is  certainly  relevant 
to  the  evaluation  of  the  prototyping  effort, 
it  automatically  introduces  biases  into  any 
evaluation  assessment  of  the  usefulness  of 
the  telemedicine  system  as  a  tool  for  health 
care.  If  the  system  functions  poorly,  clini¬ 
cians  will  cease  to  use  it  and  their  active 
assessment  of  its  utility  as  a  health  care 
tool  will  be  negative. 

Care  must  be  taken  in  choosing  opera¬ 
tional  environments  for  rapid  prototyping 
testbeds  to  ensure  that  sufficient  caseloads 

exist  to  ad- 

''These  findings  will  equately  test  the 

inevilably  lend  to  prototype  sys 

improved  use  of  to  en- 

informnlion  technol-  sure  that  the 

ogy  and  advanced  prototype  tech- 

disgnostics  in  health  nology  is  suffi- 

care  as  well  as  ciently  reliable 

improved  systems  so  that  the  re- 

that  belter  meet  the  of  gyalua- 

needs  of  military  tion  of  thebusi- 

and  ultimately,  ness  process  (in 

Civilian  users  of  ,  .  ^  , 

telemedicine  and  this  case  tele¬ 
medical  informatics  medicine)  are 

capabiiilies."  not  biased  by 

poor  technical 
performance. 

The  lessons  learned  above  are  prelimi¬ 
nary  assessments,  and  could  change  as  the 
operation  and  evaluation  continues,  and 
more  detailed  findings  become  available. 
These  findings  will  inevitably  lead  to  im¬ 


proved  use  of  information  technology  and 
advanced  diagnostics  in  health  care  as  well 
as  improved  systems  that  better  meet  the 
needs  of  military  and  ultimately,  civilian 
users  of  telemedicine  and  medical 
informatics  capabilities. 


Summary  and  Conclusion _ 

The  Department  of  Defense  is  exploit¬ 
ing  rapid  prototyping  and  “rapid  integra¬ 
tion”  of  commercial  off-the-shelf  tech¬ 
nologies  to  acquire  and  field  test 
telemedicine  applications  in  support  of 
real-world  military  operations.  Primetime 
III  was  organized  by  the  Department  of 
Defense  Telemedicine  Testbed  at  Fort 
Detrick,  Maryland,  to  conduct  a  proof  of 
principle  demonstration  and  field  tests  of 
several  telemedicine  and  medical 
informatics  technologies  in  Bosnia  and 
Hungary.  Among  the  technologies  and 
functional  capabilities  selected  were  high- 
resolution  still  imaging,  full-motion  video 
teleconferencing,  digital  radiography, 
computed  tomography,  real-time  ultra 
sound,  teledentistry,  laparoscopic  tele-sur¬ 
gery,  teledentistry,  electronic  mail,  Internet 
and  World  Wide  Web  access,  and  an  inte¬ 
grated  hospital  information  system.  The 
most  widely  used  application  was  digital 
radiography:  a  radiologist  at  only  one  of 
two  deployed  hospitals  read  and  inter¬ 
preted  more  than  800  X-rays  transmitted 
via  teleradiology  in  the  first  four  months 
of  the  operation.  Initially  inadequate  train¬ 
ing,  technical  support  and  operational  re¬ 
liability  as  well  as  lack  of  caseload  con¬ 
tributed  to  low  system  use  for  still  imag¬ 
ing  and  full-motion  video  teleconsultation 
systems.  Medical  informatics  systems  in- 
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eluding  a  deployable  version  of  the  DoD 
CHCS  and  a  prototype  PARRTS  were  suc¬ 
cessfully  implemented  and  favorably  re¬ 
ceived  by  users. 

An  important  element  of  the  Primetime 
III  augmentation  is  the  evaluation  of  clini¬ 
cal  and  technical  effectiveness  of  the  de¬ 
ployed  systems.  Prior  to  deployment,  a 
comprehensive  evaluation  plan  that  estab¬ 
lishes  performance  metrics  and  a  strategy 
to  measure  clinical  and  technical  effective¬ 
ness  and  efficiency  was  designed. 

Among  the  initial  lessons  learned  were 
that  the  DoD  Telemedicine  Testbed  needs 
to  implement  more  structure  into  its  rapid 
prototyping  and  rapid  integration  process 
if  it  continues  to  exploit  real-world  mili¬ 
tary  deployments  as  telemedicine  testbeds. 
The  DoD  needs  more  flexible  and  more 
effective  policies  and  procedures  for 
shared  use  of  telecommunications  band¬ 
width  among  medical  and  nonmedical 
users  if  telemedicine  is  to  be  effectively 
integrated  into  the  military  health  service 
system  without  stovepiped,  dedicated 
communications.  The  importance  of  clini¬ 
cal  and  technical  training  and  establish¬ 
ing  uniform  telemedicine  consultation 
protocols  and  procedures  at  both  the  send¬ 
ing  and  receiving  ends  cannot  be  over¬ 
stated.  Telemedicine  changes  both  the  way 
medicine  is  practiced  and  the  way  health 
care  is  provided  logistically;  this  cannot 
be  achieved  without  well-defined  business 
practices  and  adequate  training.  Likewise, 
adequate  technical  support  and  logistical 
sustainment  must  be  provided  if  physi¬ 
cians  are  to  develop  enough  confidence 
in  telemedicine  systems  to  rely  on  them. 
Systems  that  don’t  work  won’t  be  used. 
Even  in  a  rapid  prototyping,  proof  of  prin¬ 


ciple  operation,  adequate  planning  and 
procedures  for  providing  user  training  and 
sustainment  support  must  be  implemented 
if  patient  care  is  involved. 

The  observations  presented  in  this  pa¬ 
per  are  only  preliminary  ones.  It  is  too 
early  to  comprehensively  answer  the 
evaluation  questions  constructed  for  the 
Primetime  III  effort.  Nonetheless,  it  is 
clear  that  the  deployment  in  Bosnia  will 
provide  the  U.S.  military  health  service 
system  with  valuable  information  about 
the  appropriate  use  of  emerging 
telemedicine  and  medical  informatics 
technologies  in  support  of  military  opera¬ 
tions.  We  are  learning  that  rapid 
prototyping  and  matrixed  organizational 
approaches  that 


leverage  dis¬ 
tributed  exper¬ 
tise  while  mini¬ 
mizing  organi¬ 
zational  mass 


"Telemedicine 
changes  both  the 
way  medicine  is 
practiced  and  the 
way  health  care  is 


can  be  effective  provided  logistically; 
at  responding  to  this  cannot  be 
rapidly  chang-  achieved  without 
ing  require-  well-defined  busi- 
ments  Con  "ess  practices  ond 
versely,  we  are  "^equate  training." 

also  learning 

how  important  it  is  to  really  understand 
user  requirements  and  organizational 
change,  if  technology  use  is  meet  expec¬ 
tations.  Most  important,  we  have  learned 
that  it  is  easy  to  underestimate  the  work 
required  to  manage  human  factors  that 
ultimately  determine  whether  tele-medi¬ 
cine  will  be  used  to  its  full  advantage  to 
improve  access  to  care,  decrease  the  cost 
of  delivery,  and  improve  quality. 
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Technical  Appendix 


Task  Force  Primetime  (TFPT)  III  ca¬ 
pabilities  are  implemented  in  an  iterative- 
phased  process.  When  complete,  the  aug¬ 
mentation  will  provide  an  integrated 
telemedicine  network  for  Task  Force 
Eagle  medical  units.  The  network  will  in¬ 
clude  the  LRMC,  the  67th  CSH,  the  212th 
MASH,  2  Brigade  Operating  Base  (BOB) 
medical  units  (possibly  27),  and  access  to 
the  USS  George  Washington  in  the 
Adriatic  Sea,  off  the  coast  of  Bosnia. 

The  high  data  rate  communications  net¬ 
work  will  be  capable  of  supporting: 

a.  High-resolution,  diagnostic  quality, 
remote  video  telemedical  consulta¬ 
tions. 

b.  Remote  specialty  surgery  mentoring 
to  MASH. 

c.  Digital  diagnostic  scopes. 

d.  High-resolution,  color,  still  image 
capture. 

e.  CHCS  access. 

f.  TAMMIS. 

g.  Clinical  E-Mail. 

h.  Teleradiology. 

i.  CT  scanning. 

j.  Full-motion,  real-time  ultrasound. 

k.  CR  filmless  radiology. 


l.  Teledentistry. 

m.  Telepsychiatry. 

n.  Medical  situational  awareness. 

o.  Preventive  medicine  consulta¬ 
tions. 

p.  Infectious  disease  consultations. 

q.  Internet  access. 

At  the  discretion  of  the  Task  Force 
Eagle  Commander,  the  integrated  network 
(Figure  1 .)  also  allows  for  access  to  fixed 
facility,  tri-service  medical  centers  in  the 
United  States.  The  network  will  also  be 
designed  to  be  capable  of  expansion  and 
integration  with  the  support  of  additional 
Air  Force,  Navy,  and  Allied  forces  medi¬ 
cal  units  and  leading  edge  technology  in¬ 
sertions  by  DARPA,  as  necessary. 


Voice 


Each  location  is  equipped  with  a  Micom 
Marathon  switch  and  two  telephones. 
Voice  connectivity  between  the  TFPT 
III  locations  is  provided  by  three  256- 
kilobits-per-second  (kbps)  frame  relay 
permanent  virtual  circuits  provisioned 
through  the  IDNX  nodes.  Connectivity 
to  the  Defense  Switched  Network 
(DSN)  is  provided  by  extending  six  sub¬ 
scriber  lines  from  the  DSN  switch  at 
Lanstuhl  to  Foreign  Exchange  Office 
(FXO)  modules  in  the  Micom  switch  at 
the  LRMC.  The  six  DSN  lines  shall  be 
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configured  for  Dual-Tone  Multifrequent 
(DTMF)  signaling. 


Video 


Minimum  of  two  VTC  units  are  pro¬ 
vided  at  each  field  location.  VTC  Unit  One 
serves  the  clinic,  and  VTC  Unit  Two 
serves  the  Operating  Room  (OR).  The 
VTC  units  installationed  in  the  OR  have 
been  equipped  with  special  camera  units 
and  wireless  microphones.  The  VTC  units 
at  the  Taszar  CSH  and  Tuzla  MASH  are 
connected  to  ports  A1  on  two  separate 
Teleos  Model  20/A  1T/2P  Access  switches, 
which,  in  turn,  are  connected  to  the  net¬ 
work  through  one  Primary  Rate  Interface 
(PRI)  network  interface  each  to  the  IDNX. 
The  third  Teleos  Access  switch  installed 
at  Taszar  and  Tuzla  are  spares.  At  the 
LRMC,  two  Teleos  Model  MCU-1T/16B/ 
2P/4M  Video  Hubs  are  installed  which, 
in  addition  to  VTC  interfaces,  also  have 
four  Multipoint  Control  Unit  (MCU) 
ports,  which  will  be  used  to  set  up 
multipoint  conferences. 

VTC  equipment  from  two  manufactur¬ 
ers  is  used.  The  LRMC  and  Tuzla  MASH 
will  have  CLI  Radiance  equipment,  while 
the  Taszar  CSH  will  have  PicTel  units .  It 
should  be  noted  that  while  the  CLI  Radi¬ 
ance  units  are  capable  of  setting  up  full- 
bandwidth  conferences  at  1 .472  Mb/s  (us¬ 
ing  23  Basic  Rate  Interface  (BRI)  chan¬ 
nels),  the  PicTel  units  at  Taszar  CSH  are 
limited  to  a  maximum  of  768  kbps  (12  BRI 
channels)  conferences. 

VTC  connectivity  between  TFPT  III 
locations  is  provided  through  the  back¬ 
bone  transmission  system  that  has  the  ca¬ 
pacity  to  accommodate  one  point-to-point 
conference  at  1.472  Mb/s  or  two  point- 


to-point  conferences  at  768  kbps  between 
any  two  field  locations. 

Off-net  access  to  the  German  commer¬ 
cial  network  will  be  provided  at  Landstuhl 
by  leasing  12  BRI  Euro-ISDN  lines  from 
the  Deutsche  Bundespost  (DBP).  This 
access  will  be  used  to  arrange  conferences 
between  TFPT  III  locations  and  Heidel¬ 
berg,  Wuerzburg  or  U.S.European  Com¬ 
mand  (USEUCOM). 

Off-net  connectivity  to  the  San  Anto¬ 
nio  MEDNET  gateway  will  be  established 
through  a  384  kbps  line  (to  be  increased 
to  1.536  Mbps  at  a  future  date). 

Multipoint  conferences  can  be  arranged 
for  up  to  four  locations  using  one  of  the 
two  Teleos  MCU-1T/16B/2P/4M  units 
installed  at  the  LRMC  or  for  up  to  six  lo¬ 
cations  by  connecting  the  two  MCUs  to¬ 
gether  through  one  pair  of  the  MCU  ports, 
leaving  three  ports  on  each  unit  for  VTC 
locations. 


Data  Communications 


A  local  area  network  (LAN)  serving 
teleradiology  provided  access  to  a  high¬ 
speed  (1.536  Mb/s)  frame  relay  network 
through  a  Cisco  router  located  in  the  com¬ 
munications  equipment  area  and  a  LAN 
hub  located  in  the  teleradiology  equipment 
room.  Frame  relay  connectivity  is  pro¬ 
vided  by  private  virtual  circuits  (PVCs) 
with  Data  Link  Connection  Identifiers 
(DLCIs). 

A  LAN  serving  support  systems  such 
as  TAMMIS,  DMRIS,  and  e-mail  is  con¬ 
nected  to  a  LAN  hub  located  in  the  clinic 
and  provided  a  64  kbps  access  circuit  to 
the  N-level  Internet  protocol  (IP)  routing 
Network  (NIPRNET)  through  the  Cisco 
router  and  the  Integrated  Digital  Network 
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Figure  5.  Primetime  III  Network  Configuration. 
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Exchange  (IDNX).  Figure  5  shows  the 
network  configuration. 

Teleradiology 


LRMC 

Landstuhl  Regional  Medical  Center 
was  established  as  a  referral  hub  for  sec- 
ondary  reading  and  as  a  site  for  clinical 
consultation  with  telemedicine.  Equip¬ 
ment  includes  2K  diagnostic  workstations 
and  a  IK  high  luminescence  image  evalu¬ 
ation  workstation,  a  dry  laser  printer  with 
a  print  server,  and  digital  imaging  com¬ 
munication  for  medicine  (DICOM)  image 
gateway. 

67th  CSH,  Hungary 

The  67th  CSH  was  established  as  a  pri¬ 
mary  referral  center  for  images  obtained 
in  the  theater  and  diagnostic  center  for 
Hungary.  Equipment  includes  2K  diagnos¬ 
tic  workstations  and  IK  high  lumines¬ 
cence  image  evaluation  workstation,  and 
a  dry  laser  printer  with  a  print  server.  Im¬ 
ages  are  acquired  on  phosphor  CR  plates 
and  processed  in  a  laser  processor.  The 
equipment  is  connected  to  a  Synoptics  hub 
and  to  the  WAN  through  a  Cisco  router. 

212th  mash,  Bosnia 

The  212th  MASH  was  established  as  a 
diagnostic  center  for  Bosnia.  Equipment 
includes  2K  diagnostic  workstations  and 
IK  high  luminescence  image  evaluation 
workstation,  and  a  dry  laser  printer  with  a 
print  server.  Images  are  acquired  on  phos¬ 
phor  CR  plates  and  processed  in  a  laser 
processor. 

The  equipment  is  connected  to  a 
Synoptics  hub  and  to  the  WAN  through  a 


Cisco  router.  The  ultrasound  machine  is 
connected  to  the  NTSC  port  of  the 
telemedicine  codec  to  provide  real-time 
ultrasound. 

Image  Archive  and  Report  Generation 

All  images  are  archived  at  the  MASH. 
Images  are  initially  stored  on  the  9Gig 
RAID  array,  then  archived  FIFO  on  opti¬ 
cal  disc. 

Radiographic  reports  are  entered  manu¬ 
ally  by  the  radiologist  at  the  CSH.  A  se¬ 
ries  of  macros  have  been  written  to  speed 
the  process  of  report  generation  and  ap¬ 
proval.  Reports  are  telnetted  to  the  MASH 
from  the  CSH. 


Techmicai  Problems  in  Teleradiology 

Occurrences  in  the  teleradiology  sys¬ 
tem  are  designated  major  or  minor  events. 
Major  events  are  those  that  may  or  do  have 
a  direct  impact  on  patient  care.  Minor 
events  are  abnormal  occurrences  with  no 
potential  for  direct  impact  on  patient  care. 

One  major  event  had  a  direct  impact  on 
patient  care:  16  images  were  lost  from  a 
CR  study  and  the  exam  was  repeated  on 
conventional  film.  When  the  initial  16 
images  were  selected  to  merge  into  one 
study  there  was  a  lack  of  disk  space  in  the 
swap  partition  and  the  images  were  lost. 
This  has  been  prevented  by  instituting  pro¬ 
cedural  changes  until  a  larger  disk  drive 
can  be  moved  to  the  MASH. 

A  second  major  problem  has  been  in 
the  integrity  of  the  communications  link 
from  the  MASH  to  the  CSH.  This  has  been 
inoperable  for  as  long  as  72  hours.  Since 
some  of  the  voice  lines  and  all  Internet 
lines  are  carried  over  this  communications 
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link,  it  is  impossible  to  provide  support 
during  these  inoperative  periods. 

Another  major  event  was  rain  damage 
of  the  CR  Identification  Data  Terminal 
when  the  tent  housing  this  piece  collapsed 
during  a  storm.  The  equipment  has  been 
repositioned  into  an  isoshelter  to  prevent 
a  recurrence. 


There  have  been  11  reported  minor 
events.  These  usually  involve  printing  or 
delays  in  image  transmission.  They  have 
been  resolved  without  affecting  patient 
care. 
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ACQUISITION  REVIEW  QUARTERIY 
GUIDELINES  FOR  CONTRIBUTORS 


The  Acquisition  Review  Quarterly 
(ARQ)  is  a  scholarly  peer-reviewed  jour¬ 
nal  published  by  the  Defense  Acquisition 
University.  All  submissions  receive  a 
masked  review  to  ensure  impartial  evalu¬ 
ation. 

Submissions 


Submissions  are  welcomed  from  any¬ 
one  involved  in  the  Defense  acquisition 
process.  Defense  acquisition  is  defined  as 
the  conceptualization,  initiation,  design, 
development,  test,  contracting,  produc¬ 
tion,  deployment,  logistic  support,  modi¬ 
fication,  and  disposal  of  weapons  and 
other  systems,  supplies,  or  services  to  sat¬ 
isfy  Defense  Department  needs,  or  in¬ 
tended  for  use  in  support  of  military  mis¬ 
sions. 

Research  Articles 


Manuscripts  should  reflect  research  or 
empirically-supported  experience  in  one 
or  more  of  the  aforementioned  areas  of 
acquisition.  Research  or  tutorial  articles 
should  not  exceed  4,500  words.  Opinion 
pieces  should  be  limited  to  1,500  words. 

Research  articles  are  characterized  by 
a  systematic  inquiry  into  a  subject  to  dis¬ 
cover  or  revise  facts,  theories,  etc. 


Manuscript  Sections 


The  introduction  should  state  the  pur¬ 
pose  of  the  article  and  concisely  summa¬ 
rize  the  rationale  for  the  undertaking. 

The  methods  section  should  include  a 
detailed  methodology  that  clearly  de¬ 
scribes  work  performed.  Although  it  is  ap¬ 
propriate  to  refer  to  previous  publications 
in  this  section,  the  author  should  provide 
enough  information  so  that  the  experi¬ 
enced  reader  need  not  read  earlier  works 
to  gain  understanding  of  the  methodology. 

The  results  section  should  concisely 
summarize  findings  of  the  research  and 
follow  the  train  of  thought  established  in 
the  methods  section.  This  section  should 
not  refer  to  previous  publications,  but 
should  be  devoted  solely  to  the  current 
findings  of  the  author. 

The  discussion  section  should  empha¬ 
size  the  major  findings  of  the  study  and 
its  significance.  Information  presented  in 
the  aforementioned  sections  should  not  be 
repeated. 

Research  Considerations 


Contributors  should  also  consider  the 
following  questions  in  reviewing  their  re- 
search-based  articles  prior  to  submission: 

•  Is  the  research  question  significant? 
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•  Are  research  instruments  reliable 
and  valid? 

•  Are  outcomes  measured  in  a  way 
clearly  related  to  the  variables  un¬ 
der  study? 

•  Does  the  research  design  fully  and 
unambiguously  test  the  hypothesis? 

•  Did  you  build  needed  controls  into 
the  study? 

Contributors  of  research-based  submis¬ 
sions  are  also  reminded  they  should  share 
any  materials  and  methodology  necessary 
to  verify  their  conclusions. 

Criteria  For  Tutorials _ 

Tutorials  should  provide  special  in¬ 
struction  or  knowledge  relevant  to  an  area 
of  defense  acquisition  to  inform  the  De¬ 
fense  Acquisition  Workforce. 

Topics  for  submissions  should  rely  on 
or  be  derived  from  observation  or  experi¬ 
ment,  rather  than  theory.  The  submission 
should  provide  knowledge  in  a  particular 
area  for  a  particular  purpose. 

Opinion  Criteria _ 

Opinion  articles  should  reflect  judg¬ 
ments  based  on  the  special  knowledge  of 
the  expert.  Opinion  articles  should  be 
based  on  observable  phenomena  and  pre¬ 
sented  in  a  factual  manner;  that  is,  sub¬ 
missions  should  imply  detachment.  The 
observation  and  judgment  should  not  re¬ 
flect  the  author’s  personal  feelings  or 
thoughts.  Nevertheless,  opinion  pieces 


should  clearly  express  a  fresh  point  of 
view,  rather  than  negatively  criticize  the 
view  of  another  previous  author. 

Manuscript  Style _ 

We  will  require  you  to  recast  your  last 
version  of  the  manuscript,  especially  ci¬ 
tations  (e.g.,  footnotes  or  endnotes)  into 
the  format  required  in  two  specific  style 
manuals.  The  ARQ  follows  the  author 
(date)  form  of  citation.  We  expect  you  to 
use  the  Publication  Manual  of  the  Ameri¬ 
can  Psychological  Association  (4th  Edi¬ 
tion),  and  the  Chicago  Manual  of  Style 
(14th  Edition).  The  ARQ  follows  the  au¬ 
thor  (date)  form  of  citation. 

Contributors  are  encouraged  to  seek  the 
advice  of  a  reference  librarian  in  complet¬ 
ing  citations  of  government  documents. 
Standard  formulas  of  citations  may  give 
only  incomplete  information  in  reference 
to  government  works.  Helpful  guidance 
is  also  available  in  Gamer,  D.L.  and  Smith, 
D.H.,  1993,  The  Complete  Guide  to  Cit¬ 
ing  Government  Documents:  A  Manual 
for  Writers  and  Librarians  (Rev.  Ed.), 
Bethesda,  MD:  Congressional  Informa¬ 
tion  Service,  Inc. 


Copyright  Information _ 

The  ARQ  is  a  publication  of  the  United 
States  Government  and  as  such  is  not 
copyrighted.  Contributors  of  copyrighted 
works  and  copyright  holders  of  works  for 
hire  are  strongly  encouraged  to  request 
that  a  copyright  notification  be  placed  on 
their  published  work  as  a  safeguard  against 
unintentional  infringement.  The  work  of 
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federal  employees  undertaken  as  part  of 
their  official  duties  is  not  subject  to  copy¬ 
right. 

In  citing  the  work  of  others,  it  is  the 
contributor’s  responsibility  to  obtain  per¬ 
mission  from  a  copyright  holder  if  the  pro¬ 
posed  use  exceeds  the  fair  use  provisions 
of  the  law  (see  U.S.  Government  Printing 
Office,  1994,  Circular  92:  Copyright  Law 
of  the  United  States  of  America,  p.  15, 
Washington,  DC:  Author).  Contributors 
will  be  required  to  submit  a  copy  of  the 
written  permission  to  the  editor  before 
publication. 

Manuscript  Format 


Pages  should  be  double-spaced  and  or¬ 
ganized  in  the  following  order:  title  page, 
abstract,  body,  reference  list,  author’s  note 
(if  any),  and  figures  or  tables.  Figures  or 
tables  should  not  be  inserted  (or  embed¬ 
ded,  etc.)  into  the  text,  but  segregated  one 
to  a  page  following  the  text.  If  material  is 
submitted  on  a  computer  diskette,  each 
figure  or  table  should  be  recorded  in  a 
separate,  exportable  file  (i.e.,  a  readable 
.eps  file).  For  additional  information  on 
the  preparation  of  figures  or  tables,  see 
CBE  Scientific  Illustration  Committee, 
1988,  Illustrating  Science:  Standards  for 
Publication,  Bethesda,  MD:  Council  of 
Biology  Editors,  Inc.  Please  restructure 
briefing  charts  and  slides  to  a  look  similar 
to  those  in  previous  issues  of  ARQ. 


The  author  (or  corresponding  author  in 
the  case  of  multiple  authorship)  should 
attach  to  the  manuscript  a  signed  cover 
letter  that  provides  the  author’s  name,  ad¬ 
dress,  and  telephone  number  (fax  and 
Internet  addresses  are  also  appreciated). 
The  letter  should  verify  that  the  submis¬ 
sion  is  an  original  product  of  the  author; 
that  it  has  not  been  published  before;  and 
that  it  is  not  under  consideration  by  an¬ 
other  publication.  Details  about  the  manu¬ 
script  should  also  be  included  in  this  let¬ 
ter:  for  example,  its  title,  word  length,  the 
need  for  copyright  notification,  the  iden¬ 
tification  of  copyrighted  material  for 
which  permission  must  be  obtained,  a  de¬ 
scription  of  the  computer  application  pro¬ 
grams  and  file  names  used  on  enclosed 
diskettes,  etc. 

The  letter,  one  copy  of  the  printed 
manuscript,  and  any  diskettes  should  be 
sturdily  packaged  and  mailed  to:  Defense 
Systems  Management  College,  Attn: 
DSMC  Press  {ARQ),  9820  Belvoir  Road, 
Suite  G38,  Fort  Belvoir,  VA  22060-5565. 

In  most  cases,  the  author  will  be  noti¬ 
fied  that  the  submission  has  been  received 
within  48  hours  of  its  arrival.  Following 
an  initial  review,  submissions  will  be  re¬ 
ferred  to  referees  and  subsequent  consid¬ 
eration  by  the  ARQ  Editorial  Board. 

Contributors  may  direct  their  questions 
to  the  Editor,  ARQ,  at  the  address  shown 
above,  by  calling  (703)  805-4290  (fax 
805-2917),  or  via  the  Internet  at 
gonzalezd@dsmc.dsm.mil. 

The  DSMC  Home  Page  can  be  accessed 
at:  http://www.dsmc.dsm.mil. 
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Call  for  Papers 

Acquisition  Review  Quarterly 

Special  Issue  on  Managing  Radical  Change 


Tht  Acquisition  Review  Quarterly  {ARQ) 
is  planning  a  special  issue  with  the  theme 
“Managing  Radical  Change.”  We  are 
pleased  to  issue  this  call  for  papers! 


even  marketing  is  involved  with  some 
form  or  stage  of  the  acquisition  process. 


Although  ihtARQ  generally  maintains  an 
explicit  focus  on  the  Department  of  De¬ 
fense,  one  purpose  of  this  special  issue  is 
to  reach  beyond  its  usual  pool  of  research¬ 
ers — to  seek  scholarly  theoretical  contri¬ 
butions  and  well-researched  empirical 
papers  of  a  basic  and  exploratory  nature 
that  may  not  be  strictly  focused  on  defense 
acquisition.  The  term  acquisition  refers  to 
the  enterprise  processes  required  for  plan¬ 
ning,  procuring,  implementing,  managing, 
and  maintaining  major  systems;  this  might 
include,  for  example,  information  systems, 
factory  equipment,  transportation  and 
communication  networks,  intelligence  and 
research  processes,  and  other  significant 
systems,  in  addition  to  defense  weaponry. 
Every  enterprise  that  engages  in  strategic 
and  capital  planning,  procurement,  project 
management,  policy  making,  logistics, 
systems  engineering,  operations  manage¬ 
ment,  outsourcing,  performance  measure¬ 
ment,  decision  making,  collaboration,  and 


The  focus  of  this  special  issue  is  on  man¬ 
aging  radical  change,  which  has  roots  in 
the  literature  on  business  process 
reengineering,  change  management,  new 
public  management,  organizational  de¬ 
sign,  revolutionary  economics,  leadership, 
systems  analysis,  policy,  quality,  and  oth¬ 
ers.  This  focus  comes  at  a  time  in  which 
many  enterprises  and  organizations  have 
now  had  some  experience  with  the  kinds 
of  radical  change  and  fundamental  pro¬ 
cess  redesign  that  are  often  used  to  con¬ 
trast  reengineering  and  innovation  with  the 
more  incremental  modes  of  change  gen¬ 
erally  associated  with  Total  Quality  Man¬ 
agement  and  continuous  process  improve¬ 
ment.  The  crumbling  of  the  Berlin  Wall, 
combined  with  acquisition  reform,  tech¬ 
nological  innovation  and  downsizing 
present  tremendous  challenges  in  terms  of 
the  defense  acquisition  process,  and  one 
objective  might  be  to  draw  from  research 
focused  on  industry  to  help  address  these 
challenges. 


This  special  issue  seeks  descriptive  and 
prescriptive  theoretical  work  that  can  be 
applied  to  the  acquisition  process,  along 
with  rigorous  empirical  work  with  clear 
generalizations  and  implications  for  de¬ 
fense  acquisition.  The  ARQ  is  read  by  the 
most  senior  executives  and  policy  mak¬ 
ers  in  the  government,  so  papers  must  have 
sufficient  impact  and  importance  to  inter¬ 
est  this  group.  We  also  seek  to  build  a  more 
solid  theoretical  foundation  upon  which 
acquisition  research  can  build,  so  papers 
should  be  well-grounded  in  the  appropri¬ 
ate  literature,  presented  and  formalized  in 
a  clear,  concise,  convincing  and  scientifi- 
cally-defensible  manner,  and  they  should 
include  compelling  routes  to  generaliza¬ 
tion,  replication,  falsification  and  exten¬ 
sion. 

The  ARQ  is  a  refereed  journal  that  is  spon¬ 
sored  by  the  Department  of  Defense,  and 
represents  the  leading  source  of  executive 
and  policy  guidance  pertaining  to  defense 
acquisition.  This  special  issue  will  include 
professors  David  Lamm,  Mark  Nissen  and 
Keith  Snider,  from  the  Naval  Postgradu¬ 
ate  School,  as  Guest  Editors.  Refereeing 
will  be  conducted  in  the  usual  double¬ 
blind  process.  Authors  wishing  to  submit 
manuscripts  should  send  them  to  the  Edi¬ 
tor  of  the  ARQ,  at  the  address  below  not 
later  than  31  July  1997. 

Manuscripts  will  be  initially  screened  by 
the  Editor  of  the  ARQ  to  determine  their 
appropriateness  for  review,  and  should  not 
exceed  6000  words.  To  ensure  anonym¬ 


ity,  each  paper  should  be  submitted  with 
a  separate  page  that  includes  the 
author(s)’s  name(s)  and  complete  address, 
and  the  paper  should  include  the  title,  ab¬ 
stract,  keywords,  body,  complete  set  of  ref¬ 
erences,  along  with  tables  and  figures  at 
the  end.  Authors  are  reminded  to  not  refer 
to  themselves  or  to  their  own  work  directly 
in  the  paper.  All  ARQ  publications  must 
follow  the  Publication  Manual  of  the 
American  Psychological  Association  for¬ 
mat,  and  potential  authors  may  wish  to 
consult  a  recent  issue  of  the  Journal  for 
reference. 

If  you  have  preliminary  questions  on 
submissions  to  this  ARQ  Special  Issue, 
contact  these  Guest  Editors: 

Dr.  David  Lamm 

(DLamm@nps.navy.mil;  408-656-2775) 
Dr.  Mark  Nissen 

(MNissen@nps.navy.mil;  408-656-3570) 
Prof.  Keith  Snider 

(KSnider@nps,navy.mil;  408-656-3621) 

Naval  Postgraduate  School 
555  Dyer  Road,  Code  SM 
Monterey,  CA  93943-5000 

To  submit  papers,  mail  them  to: 
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9820  Belvoir  Road,  Suite  G38 
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